Oct 29 22:39:44.916157 ip-10-0-136-187 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 29 22:39:44.916170 ip-10-0-136-187 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 29 22:39:44.916180 ip-10-0-136-187 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 29 22:39:44.916585 ip-10-0-136-187 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 29 22:39:55.165023 ip-10-0-136-187 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 29 22:39:55.165043 ip-10-0-136-187 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot f6dd021f7c1f47f5af6232f447ef1529 --
Oct 29 22:41:40.968295 ip-10-0-136-187 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:41:41.349857 ip-10-0-136-187 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:41:41.349857 ip-10-0-136-187 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:41:41.349857 ip-10-0-136-187 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:41:41.349857 ip-10-0-136-187 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:41:41.354620 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354527    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.354620 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354616    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.354620 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354621    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.354620 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354626    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354630    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354636    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354641    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354646    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354649    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354653    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354657    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354661    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354665    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354679    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354684    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354688    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354692    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354696    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354700    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354705    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354709    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354713    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354717    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.354873 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354721    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354725    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354729    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354732    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354736    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354740    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354744    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354748    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354752    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354756    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354760    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354764    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354768    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354773    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354778    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354783    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354787    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354792    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354796    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354800    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.355735 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354805    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354810    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354814    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354818    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354823    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354827    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354831    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354835    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354839    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354843    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354847    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354852    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354857    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354861    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354865    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354869    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354873    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354878    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354882    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354887    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.357866 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354891    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354897    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354903    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354907    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354912    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354916    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354921    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354928    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354934    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354940    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354946    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354951    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354956    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354961    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354966    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354970    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354975    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354979    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.358457 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354984    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354990    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.354996    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355001    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355021    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355026    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355030    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355035    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355040    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355044    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355048    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355053    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355058    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355069    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355073    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355077    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355082    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355086    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355090    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.358874 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355095    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355099    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355103    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355108    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355112    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.355129    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356510    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356523    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356527    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356532    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356536    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356539    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356542    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356544    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356547    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356550    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356552    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356555    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356557    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356560    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.359708 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356563    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356566    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356568    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356571    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356574    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356577    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356580    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356583    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356585    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356588    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356590    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356593    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356596    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356599    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356601    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356604    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356606    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356609    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356612    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.360589 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356615    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356617    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356620    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356623    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356625    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356628    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356630    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356633    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356636    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356638    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356641    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356643    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356646    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356650    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356654    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356658    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356661    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356664    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356667    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.361381 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356670    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356672    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356675    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356678    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356681    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356683    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356686    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356689    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356691    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356694    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356696    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356699    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356702    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356705    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356708    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356711    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356714    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356716    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356719    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356721    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.361842 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356724    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356726    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356729    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356731    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356734    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356737    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356739    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356742    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356744    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356747    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356750    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356752    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356756    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356759    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356762    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356765    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356767    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356770    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356772    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.362378 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356775    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356778    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356780    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356783    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356785    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356788    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356791    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356796    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356799    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356803    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356806    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356809    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356812    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356814    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.356817    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357455    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357463    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357469    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357473    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.362859 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357477    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357480    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357482    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357485    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357488    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357490    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357494    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357497    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357499    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357502    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357505    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357507    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357510    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357513    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357515    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357518    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357520    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357523    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357525    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357528    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.363409 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357531    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357534    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357537    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357539    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357542    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357544    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357547    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357549    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357552    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357555    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357558    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357560    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357563    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357565    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357568    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357570    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357572    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357575    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357578    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.363928 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357583    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357590    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357595    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357600    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357603    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357609    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357615    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357619    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357623    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357627    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357630    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357634    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357638    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357642    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357647    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357651    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357655    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357659    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357663    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357667    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.364436 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357671    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357674    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357678    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357682    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357687    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357692    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357696    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357700    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357704    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357709    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357713    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357717    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357722    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357726    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357730    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357736    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357740    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357745    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357749    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357753    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.364990 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357758    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357762    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357766    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357771    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357775    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357779    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357783    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357788    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357792    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357797    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357801    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357806    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357810    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357815    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357819    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357824    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357828    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357832    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357837    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.365531 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357841    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357845    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357850    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.357854    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358896    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358914    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358926    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358933    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358941    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358947    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358954    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358961    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358966    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358973    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358979    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358985    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358990    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.358995    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359000    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359021    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359026    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359033    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359039    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:41:41.366034 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359044    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359049    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359053    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359067    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359074    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359083    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359089    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359094    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359099    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359105    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359110    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359115    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359120    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359128    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359132    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359137    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359142    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359148    2574 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359152    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359160    2574 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359165    2574 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359170    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359175    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359180    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:41:41.366622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359187    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359192    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359199    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359204    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359209    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359213    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359219    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359225    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359230    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359235    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359239    2574 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359246    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359251    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359258    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359263    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359269    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359274    2574 flags.go:64] FLAG: --help="false"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359279    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359284    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359289    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359294    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359299    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359305    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359310    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:41:41.367266 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359315    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359319    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359324    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359329    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359334    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359339    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359343    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359348    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359353    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359358    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359364    2574 flags.go:64] FLAG: --lock-file=""
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359369    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359374    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359379    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359390    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359395    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359401    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359405    2574 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359410    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359415    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359420    2574 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359427    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359435    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359440    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359447    2574 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:41:41.367847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359452    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359457    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359461    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359466    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359470    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359476    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359480    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359491    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359496    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359501    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359506    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359511    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359516    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359520    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359525    2574 flags.go:64] FLAG: --port="10250"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359530    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359535    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08079b5c878559f90"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359541    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359546    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359552    2574 flags.go:64] FLAG: --register-node="true"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359557    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359563    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359568    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359573    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359579    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359585    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:41:41.368468 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359590    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359595    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359602    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359607    2574 flags.go:64] FLAG: --runonce="false"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359612    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359617    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359622    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359627    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359632    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359637    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359642    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359647    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359652    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359656    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359661    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359665    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359671    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359676    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359680    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359689    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359694    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359699    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359705    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359711    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359715    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:41:41.369186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359720    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359724    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359729    2574 flags.go:64] FLAG: --v="2"
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359736    2574 flags.go:64] FLAG: --version="false"
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359743    2574 flags.go:64] FLAG: --vmodule=""
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359750    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.359756    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.359993    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360000    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360024    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360029    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360035    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360039    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360043    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360048    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360052    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360057    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360061    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360066    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360071    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360075    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.369823 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360079    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360084    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360088    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360093    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360098    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360102    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360107    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360111    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360116    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360122    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360126    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360131    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360135    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360140    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360144    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360149    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360153    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360158    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360171    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360176    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.370344 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360182    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360187    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360191    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360195    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360200    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360205    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360209    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360214    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360218    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360222    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360227    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360231    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360236    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360240    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360244    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360250    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360257    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360262    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360267    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.370830 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360271    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360275    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360281    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360286    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360290    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360295    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360299    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360303    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360308    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360312    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360316    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360321    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360333    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360339    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360343    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360348    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360352    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360357    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360361    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.371304 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360367    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360371    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360375    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360380    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360384    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360388    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360393    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360397    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360401    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360405    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360410    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360414    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360418    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360423    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360427    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360433    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360438    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360442    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360446    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360450    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.371747 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360454    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360459    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360463    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360467    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360471    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360478    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360486    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360491    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360496    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360500    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360504    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360508    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360513    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360517    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.360564    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360630    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.372237 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360638    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360643    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360647    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360652    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360656    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360660    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360664    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360668    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360673    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360677    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360681    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360686    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360693    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360697    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360701    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360705    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360709    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360713    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360718    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360722    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.372619 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360726    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360731    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360737    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360745    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360752    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360757    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360763    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360768    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360773    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360777    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360781    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360785    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360790    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360794    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360799    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360803    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360807    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360812    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360816    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.373106 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360820    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360824    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360829    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360833    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360837    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360844    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360849    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360853    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360857    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360864    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360869    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360873    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360877    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360882    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360886    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360892    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360897    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360902    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360906    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.373551 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360910    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360914    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360919    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360923    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360927    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360931    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360936    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360940    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360944    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360949    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360953    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360957    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360963    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360967    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360971    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360976    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360980    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360984    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360990    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360995    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.374188 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.360999    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361019    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361023    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361030    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361036    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361042    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361047    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361051    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361057    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361062    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361066    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361071    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361075    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361080    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361084    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361088    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361093    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361097    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361101    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.375021 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361105    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361110    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361114    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361119    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361123    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361127    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361132    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.361138    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.368585    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.368603    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.368621    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.368625    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368694    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368700    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368704    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368708    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368711    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368714    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368718    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.375561 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368720    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368723    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368727    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368730    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368733    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368737    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368741    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368744    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368746    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368749    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368752    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368755    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368757    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368760    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368763    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368766    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368768    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368771    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368774    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.376153 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368777    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368779    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368782    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368792    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368795    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368798    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368801    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368804    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368806    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368809    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368811    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368814    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368817    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368820    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368822    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368825    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368828    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368830    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368833    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368835    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.376635 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368838    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368840    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368843    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368846    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368848    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368851    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368853    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368856    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368859    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368861    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368864    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368866    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368869    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368871    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368874    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368877    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368891    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368894    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368897    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368899    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.377183 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368902    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368904    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368907    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368909    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368912    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368914    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368917    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368919    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368922    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368924    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368927    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368930    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368932    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368935    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368937    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368940    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368942    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368944    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368947    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368949    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.377656 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368952    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368955    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368957    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368960    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368962    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368965    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368968    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368971    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368974    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368986    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368989    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368991    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368994    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368996    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.368999    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369002    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369017    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369020    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369022    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.378127 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369025    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.369057    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369247    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369251    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369255    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369258    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369260    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369263    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369266    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369268    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369271    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369273    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369276    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369278    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369281    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369283    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.378572 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369287    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369292    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369294    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369297    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369301    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369303    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369306    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369316    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369319    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369321    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369324    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369327    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369329    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369332    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369334    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369337    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369340    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369342    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369345    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.378966 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369348    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369351    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369353    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369356    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369358    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369361    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369364    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369366    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369369    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369371    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369374    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369376    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369379    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369381    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369385    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369389    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369392    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369395    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369398    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.379488 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369401    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369404    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369412    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369415    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369418    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369420    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369423    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369425    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369428    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369431    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369433    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369436    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369438    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369440    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369443    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369446    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369449    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369451    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369454    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369456    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.379932 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369459    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369462    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369464    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369466    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369469    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369472    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369475    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369477    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369480    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369483    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369485    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369488    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369490    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369493    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369495    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369503    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369506    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369509    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369512    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.380415 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369514    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369517    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369520    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369522    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369525    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369527    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369530    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369532    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369535    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369537    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369540    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369542    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369545    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369547    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369550    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.369573    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:41:41.380879 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369609    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369613    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369616    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369618    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369621    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369624    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369626    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369629    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369632    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369634    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369641    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369643    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369646    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369654    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369657    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369660    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369663    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369665    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369668    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369670    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:41:41.381326 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369673    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369675    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369678    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369681    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369683    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369686    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369689    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369691    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369694    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369696    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369699    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369701    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369704    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369707    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369709    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369712    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369714    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369717    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369720    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369722    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:41:41.381851 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369724    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369727    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369730    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369732    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369734    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369737    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369745    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369748    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369751    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369753    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369755    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369758    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369762    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369765    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369768    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369771    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369774    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369777    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369779    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369782    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:41:41.382428 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369784    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369787    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369789    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369792    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369794    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369797    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369799    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369802    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369804    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369807    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369810    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369812    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369820    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369824    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369826    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369829    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369832    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369835    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369838    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:41:41.382940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369847    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369850    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369852    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369855    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369857    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369860    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369863    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369867    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369870    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369873    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369876    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369879    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369882    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369884    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369887    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369889    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369892    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369894    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369896    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:41:41.383407 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369899    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369902    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369905    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369907    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369910    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369912    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369918    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:41.369920    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.369983    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.375568    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.376838    2574 server.go:1014] "Starting client certificate rotation"
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.376936    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.376963    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.379488    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:41:41.383940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.380875    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:41:41.394435 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.394416    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:41:41.398525 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.398507    2574 log.go:25] "Validated CRI v1 image API"
Oct 29 22:41:41.399514 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.399493    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:41:41.402403 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.402376    2574 fs.go:135] Filesystem UUIDs: map[24898a22-b72a-4538-b907-684b669d2735:/dev/nvme0n1p4 38af0018-b430-40f9-a22f-13157c698ca9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 29 22:41:41.402462 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.402402    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 29 22:41:41.409183 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.409066    2574 manager.go:217] Machine: {Timestamp:2026-10-29 22:41:41.406547824 +0000 UTC m=+0.341723569 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098780 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ba95584ca1a013eec7ea328c58ce9 SystemUUID:ec2ba955-84ca-1a01-3eec-7ea328c58ce9 BootID:f6dd021f-7c1f-47f5-af62-32f447ef1529 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c4:83:14:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c4:83:14:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:a2:44:fd:cd:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:41:41.409183 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.409182    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:41:41.409316 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.409304    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:41:41.412626 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.412593    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:41:41.412773 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.412629    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:41:41.412814 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.412776    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:41:41.412814 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.412784    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:41:41.412875 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.412814    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:41:41.412875 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.412827    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:41:41.414063 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.414052    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:41:41.418727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.418715    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:41:41.419042 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.419032    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:41:41.419990 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.419981    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:41:41.420042 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.420002    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:41:41.420042 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.420029    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:41:41.420119 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.420055    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:41:41.420119 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.420067    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:41:41.420561 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.420549    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:41:41.424831 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.424810    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:41:41.431498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.431484    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:41:41.432411 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432398    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:41:41.432449 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432436    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:41:41.432595 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432585    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:41:41.432629 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432600    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:41:41.432629 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432606    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:41:41.432629 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432611    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:41:41.432629 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432617    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:41:41.432629 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.432623    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:41:41.433432 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.433422    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:41:41.433463 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.433433    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:41:41.433463 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.433440    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:41:41.433463 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.433446    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:41:41.433463 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.433455    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:41:41.433584 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.433464    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:41:41.434194 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.434185    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:41:41.434194 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.434194    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:41:41.438245 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.438230    2574 server.go:1263] "Started kubelet"
Oct 29 22:41:41.438384 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.438342    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:41:41.438444 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.438355    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:41:41.438492 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.438447    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:41:41.439189 ip-10-0-136-187 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:41:41.440056 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.440037    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:41:41.441302 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.441289    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:41:41.445807 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.445788    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 29 22:41:41.447490 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.447473    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:41:41.448137 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.448117    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:41:41.448235 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.448218    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:41.448917 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.448898    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:41:41.448917 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.448915    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:41:41.449070 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449057    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:41:41.449070 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449071    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:41:41.449142 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449088    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:41:41.449183 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449169    2574 factory.go:55] Registering systemd factory
Oct 29 22:41:41.449218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449210    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:41:41.449458 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449444    2574 factory.go:153] Registering CRI-O factory
Oct 29 22:41:41.449528 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449460    2574 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:41:41.449528 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449510    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:41:41.449622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449528    2574 factory.go:103] Registering Raw factory
Oct 29 22:41:41.449622 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.449553    2574 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:41:41.451188 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.451173    2574 manager.go:319] Starting recovery of all containers
Oct 29 22:41:41.464226 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.464210    2574 manager.go:324] Recovery completed
Oct 29 22:41:41.468707 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.468689    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:41.469445 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.469424    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 29 22:41:41.469528 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.469434    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 29 22:41:41.469825 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.469802    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:41:41.470630 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.470610    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:41.470706 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.470647    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:41.470706 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.470662    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:41.470795 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.469597    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f61109f39  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.438201657 +0000 UTC m=+0.373377403,LastTimestamp:2026-10-29 22:41:41.438201657 +0000 UTC m=+0.373377403,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.471308 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.471291    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:41:41.471308 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.471307    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:41:41.471420 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.471326    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:41:41.471420 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.471298    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:41.472268 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.472246    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:41.472361 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.472287    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:41.472361 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.472305    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:41.473631 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.473614    2574 policy_none.go:50] "Start"
Oct 29 22:41:41.473631 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.473633    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:41:41.473716 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.473647    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:41:41.475663 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.475652    2574 policy_none.go:44] "Start"
Oct 29 22:41:41.477134 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.477117    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phlds"
Oct 29 22:41:41.483040 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.483020    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-phlds"
Oct 29 22:41:41.484759 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.484719    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 29 22:41:41.484863 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.484734    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 29 22:41:41.484863 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.484807    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.508422    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.508507    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f62ff744c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.470630988 +0000 UTC m=+0.405806738,LastTimestamp:2026-10-29 22:41:41.470630988 +0000 UTC m=+0.405806738,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.511361    2574 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.511402    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.511414    2574 server.go:87] "Starting device plugin registration server"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.511696    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.511709    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.511790    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.511898    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.511905    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.512560    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.512610    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.518729    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.519947    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.519973    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.519993    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.520046    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.524567    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f62ffcfd3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-187.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.470654419 +0000 UTC m=+0.405830168,LastTimestamp:2026-10-29 22:41:41.470654419 +0000 UTC m=+0.405830168,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.539727    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f630004fc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-187.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.470668028 +0000 UTC m=+0.405843776,LastTimestamp:2026-10-29 22:41:41.470668028 +0000 UTC m=+0.405843776,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.609513 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.557366    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-187.ec2.internal.18e3214f62ff744c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f62ff744c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.470630988 +0000 UTC m=+0.405806738,LastTimestamp:2026-10-29 22:41:41.472268333 +0000 UTC m=+0.407444163,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.610259 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.557447    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 29 22:41:41.610259 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.574455    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-187.ec2.internal.18e3214f62ffcfd3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f62ffcfd3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-187.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.470654419 +0000 UTC m=+0.405830168,LastTimestamp:2026-10-29 22:41:41.472295872 +0000 UTC m=+0.407471619,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.610259 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.601119    2574 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-187.ec2.internal.18e3214f630004fc\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f630004fc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-187.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.470668028 +0000 UTC m=+0.405843776,LastTimestamp:2026-10-29 22:41:41.472311542 +0000 UTC m=+0.407487289,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.612057 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.612037    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:41.612934 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.612912    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:41.613057 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.612947    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:41.613057 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.612960    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:41.613057 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.612986    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.617714 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.617634    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-187.ec2.internal.18e3214f656c72bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-187.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-136-187.ec2.internal,},FirstTimestamp:2026-10-29 22:41:41.511328447 +0000 UTC m=+0.446504194,LastTimestamp:2026-10-29 22:41:41.511328447 +0000 UTC m=+0.446504194,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-187.ec2.internal,}"
Oct 29 22:41:41.621018 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.620978    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"]
Oct 29 22:41:41.621100 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.621054    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:41.622804 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.622789    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:41.622890 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.622815    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:41.622890 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.622824    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:41.625108 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625096    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:41.625269 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625253    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.625313 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625292    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:41.625790 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625774    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:41.625790 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625783    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:41.625898 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625803    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:41.625898 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625808    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:41.625898 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625817    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:41.625898 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.625820    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:41.627998 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.627974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.628089 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.628003    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:41.628799 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.628781    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:41.628868 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.628812    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:41.628868 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.628827    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:41.654880 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.654860    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.654980 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.654884    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-187.ec2.internal\": node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:41.663178 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.663155    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-187.ec2.internal\" not found" node="ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.665361 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.665343    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-187.ec2.internal\" not found" node="ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.704102 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.704080    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:41.749941 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.749915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a74b888ca6cfc8b230ed1966a77b369-config\") pod \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" (UID: \"5a74b888ca6cfc8b230ed1966a77b369\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.750082 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.749952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.750082 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.749972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.804698 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.804672    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:41.850264 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.850239    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.850312 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.850269    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.850312 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.850290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a74b888ca6cfc8b230ed1966a77b369-config\") pod \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" (UID: \"5a74b888ca6cfc8b230ed1966a77b369\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.850382 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.850331    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.850382 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.850337    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.850443 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.850399    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a74b888ca6cfc8b230ed1966a77b369-config\") pod \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" (UID: \"5a74b888ca6cfc8b230ed1966a77b369\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.905780 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:41.905728    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:41.967918 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.967895    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:41.970951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:41.970933    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:42.006263 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:42.006231    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:43.824895 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:43.824867    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:43.925484 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:43.925409    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.025955 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:44.025923    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.094469 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.094438    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 29 22:41:44.126733 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:44.126707    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.226990 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:44.226795    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.277814 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:44.277785    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f8c8d4e5e3caee624106946868ca7a.slice/crio-2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516 WatchSource:0}: Error finding container 2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516: Status 404 returned error can't find the container with id 2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516
Oct 29 22:41:44.278515 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:44.278496    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a74b888ca6cfc8b230ed1966a77b369.slice/crio-d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3 WatchSource:0}: Error finding container d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3: Status 404 returned error can't find the container with id d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3
Oct 29 22:41:44.286116 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.286089    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:41:44.327961 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:44.327922    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.328095 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.327930    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:43 +0000 UTC" deadline="2028-05-06 23:33:47.334232586 +0000 UTC"
Oct 29 22:41:44.328095 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.328005    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13320h52m3.006229839s"
Oct 29 22:41:44.372350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.372330    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:41:44.373565 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.373548    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:41:44.373671 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.373578    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:41:44.373671 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.373587    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:41:44.428906 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:44.428882    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.441747 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.441726    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:44.529283 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:44.529189    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.630102 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:44.630066    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:41:44.632655 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.632630    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:44.637770 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.637736    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:44.667443 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.667409    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:44.680560 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.680534    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:41:44.687834 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.687810    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:41:44.688826 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.688765    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:41:44.704435 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:44.704400    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:41:45.139083 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.139061    2574 apiserver.go:52] "Watching apiserver"
Oct 29 22:41:45.161248 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.161214    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:41:45.161641 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.161612    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6ltpj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal","openshift-multus/multus-additional-cni-plugins-w2wcc","kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal","openshift-monitoring/node-exporter-6jfhv","openshift-network-diagnostics/network-check-target-m7w8j","kube-system/konnectivity-agent-dwfrq","openshift-ovn-kubernetes/ovnkube-node-z5wst","openshift-multus/network-metrics-daemon-6nqg4","openshift-network-operator/iptables-alerter-ltvq9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r","openshift-multus/multus-zksnr","openshift-image-registry/node-ca-qxjxd","kube-system/global-pull-secret-syncer-cn2r6","openshift-dns/node-resolver-rhshl"]
Oct 29 22:41:45.164328 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.164293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.166871 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.166331    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 29 22:41:45.166871 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.166474    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.168798 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.168779    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.171267 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.171250    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.173237 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.173211    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.175493 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.175473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:45.175592 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.175567    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:45.177062 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-registration-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.177171 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177075    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtcx\" (UniqueName: \"kubernetes.io/projected/0893ab11-598b-4072-893b-720af09698a7-kube-api-access-jrtcx\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.177171 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177102    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-cnibin\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.177171 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177127    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2abeb81-5999-493f-9474-8bf5d01b32df-iptables-alerter-script\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.177329 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177274    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7jzk\""
Oct 29 22:41:45.177418 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177399    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:41:45.177554 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177535    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.177816 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2qc\" (UniqueName: \"kubernetes.io/projected/d2abeb81-5999-493f-9474-8bf5d01b32df-kube-api-access-8f2qc\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.177897 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177834    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-device-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.177897 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177858    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.177897 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.178056 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177905    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.178056 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.178056 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-socket-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.178056 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.177988    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-sys-fs\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.178056 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178015    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.178293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178091    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pdbx4\""
Oct 29 22:41:45.178293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-system-cni-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.178293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.178293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6s75\" (UniqueName: \"kubernetes.io/projected/f7a33c42-a915-4f9c-bd12-311c839696f5-kube-api-access-n6s75\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.178293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178218    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:41:45.178293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2abeb81-5999-493f-9474-8bf5d01b32df-host-slash\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.178293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178253    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.178631 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178326    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.178631 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.178358    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.179847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.179738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.179847 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.179808    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:45.183406 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.183380    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:45.183500 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.183453    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:45.186749 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.186721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.189608 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.189552    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.191750 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.191722    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.194184 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.194163    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.196439 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.196421    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.205065 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.205041    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 29 22:41:45.205171 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.205106    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5t9b8"
Oct 29 22:41:45.211704 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.211630    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5t9b8"
Oct 29 22:41:45.242697 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.242639    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal" event={"ID":"5a74b888ca6cfc8b230ed1966a77b369","Type":"ContainerStarted","Data":"d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3"}
Oct 29 22:41:45.244364 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.244334    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal" event={"ID":"a7f8c8d4e5e3caee624106946868ca7a","Type":"ContainerStarted","Data":"2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516"}
Oct 29 22:41:45.268453 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.268429    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:41:45.277640 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.277606    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:41:45.277757 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.277612    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.277842 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.277807    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:41:45.277910 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.277880    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.277971 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.277933    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:41:45.277971 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.277942    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:41:45.277971 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.277970    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:41:45.278138 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278025    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56msd\""
Oct 29 22:41:45.278138 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278060    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.278138 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278082    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:41:45.278138 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278128    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.278138 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278169    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g5fdr\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278187    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278246    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-528cc\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278255    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278363    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278367    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278385    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6s75\" (UniqueName: \"kubernetes.io/projected/f7a33c42-a915-4f9c-bd12-311c839696f5-kube-api-access-n6s75\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278409    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.278434 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278434    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnfpb\" (UniqueName: \"kubernetes.io/projected/782ad8cb-0365-4cb1-86b0-4d3059401b91-kube-api-access-lnfpb\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.278800 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-sys\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.278800 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-etc-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.278800 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278525    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-cni-binary-copy\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.278800 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-registration-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.278800 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtcx\" (UniqueName: \"kubernetes.io/projected/0893ab11-598b-4072-893b-720af09698a7-kube-api-access-jrtcx\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.278800 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278689    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-registration-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.278800 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278791    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782ad8cb-0365-4cb1-86b0-4d3059401b91-hosts-file\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.279054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-kubelet-config\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.279054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278848    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gqfpm\""
Oct 29 22:41:45.279054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-run\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.278967    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-lib-modules\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q9qw\" (UniqueName: \"kubernetes.io/projected/182f9264-f98f-401d-aa1d-001f404fc6af-kube-api-access-2q9qw\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.279054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-hostroot\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.279285 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-device-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.279285 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279119    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.279285 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.279285 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279215    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-device-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.279285 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279285 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279247    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-netd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.279285 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279271    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279303    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-systemd\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-tmp\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/182f9264-f98f-401d-aa1d-001f404fc6af-ovn-node-metrics-cert\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279378    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-multus\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqhg\" (UniqueName: \"kubernetes.io/projected/8936c93b-d868-46be-83e4-b6d224d4b11c-kube-api-access-dkqhg\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-socket-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279519    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-conf\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-etc-tuned\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279587 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279608    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-socket-dir-parent\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279627    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-socket-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279629    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-etc-kubernetes\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-var-lib-kubelet\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-systemd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279720    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-agent-certs\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2abeb81-5999-493f-9474-8bf5d01b32df-iptables-alerter-script\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.279951 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279947    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.279997    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782ad8cb-0365-4cb1-86b0-4d3059401b91-tmp-dir\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280024    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-cnibin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-node-log\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280076    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-log-socket\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280102    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-sys-fs\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-system-cni-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280155    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5jh\" (UniqueName: \"kubernetes.io/projected/683fd3fa-45a0-429a-9146-237713d74f5d-kube-api-access-rn5jh\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280173    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-systemd-units\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280185    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-sys-fs\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280191    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-daemon-config\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-multus-certs\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280223    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-system-cni-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2abeb81-5999-493f-9474-8bf5d01b32df-host-slash\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280313    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2abeb81-5999-493f-9474-8bf5d01b32df-host-slash\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.280392 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280306    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-root\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280365    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2abeb81-5999-493f-9474-8bf5d01b32df-iptables-alerter-script\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280387    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-metrics-client-ca\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-env-overrides\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-system-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280513    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-konnectivity-ca\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280538    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280581    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-tls\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280646    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-sys\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-host\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-textfile\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280792    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-wtmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280817    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-dbus\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-kubernetes\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.281966 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-slash\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280920    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-netns\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-ovn\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.280990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281015    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-bin\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-config\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281108    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-script-lib\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281131    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-kubelet\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281183    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3ec7504-ce41-4598-b6e6-1c8706119621-host\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281188    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281214    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6lw5\" (UniqueName: \"kubernetes.io/projected/e3ec7504-ce41-4598-b6e6-1c8706119621-kube-api-access-f6lw5\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281260    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281282    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-cnibin\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281299    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-modprobe-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysconfig\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.282588 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-kubelet\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-var-lib-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2qc\" (UniqueName: \"kubernetes.io/projected/d2abeb81-5999-493f-9474-8bf5d01b32df-kube-api-access-8f2qc\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vmp\" (UniqueName: \"kubernetes.io/projected/6ac72c66-33a8-476a-afc7-093dc998aa4e-kube-api-access-46vmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-k8s-cni-cncf-io\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281489    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-netns\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-bin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281530    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-conf-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjph9\" (UniqueName: \"kubernetes.io/projected/eaf49b15-48f0-4abc-8f46-786db6035d5c-kube-api-access-bjph9\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3ec7504-ce41-4598-b6e6-1c8706119621-serviceca\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281799    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.283218 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.281826    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-cnibin\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.287273 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287252    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:41:45.287365 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287311    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287643    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287691    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287708    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qj89s\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287730    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kfh6g\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287758    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287781    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287808    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287826    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmngh\""
Oct 29 22:41:45.287874 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.287848    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:41:45.288510 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.288222    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:41:45.288510 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.288259    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bd868\""
Oct 29 22:41:45.288510 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.288274    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:41:45.288510 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.288324    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:41:45.289813 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.289793    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.290916 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.290888    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.291359 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.291338    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.294281 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.294261    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:41:45.299319 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.299291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6s75\" (UniqueName: \"kubernetes.io/projected/f7a33c42-a915-4f9c-bd12-311c839696f5-kube-api-access-n6s75\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.299691 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.299670    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtcx\" (UniqueName: \"kubernetes.io/projected/0893ab11-598b-4072-893b-720af09698a7-kube-api-access-jrtcx\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.300006 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.299989    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2qc\" (UniqueName: \"kubernetes.io/projected/d2abeb81-5999-493f-9474-8bf5d01b32df-kube-api-access-8f2qc\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.382872 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.382835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-conf-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.382886    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjph9\" (UniqueName: \"kubernetes.io/projected/eaf49b15-48f0-4abc-8f46-786db6035d5c-kube-api-access-bjph9\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.382911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3ec7504-ce41-4598-b6e6-1c8706119621-serviceca\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.382921    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-conf-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.382936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfpb\" (UniqueName: \"kubernetes.io/projected/782ad8cb-0365-4cb1-86b0-4d3059401b91-kube-api-access-lnfpb\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.382959    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-sys\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.382981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-etc-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-cni-binary-copy\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383024    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782ad8cb-0365-4cb1-86b0-4d3059401b91-hosts-file\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.383054 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-kubelet-config\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-run\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-lib-modules\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383110    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9qw\" (UniqueName: \"kubernetes.io/projected/182f9264-f98f-401d-aa1d-001f404fc6af-kube-api-access-2q9qw\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-hostroot\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383172    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383241    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-netd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383244    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782ad8cb-0365-4cb1-86b0-4d3059401b91-hosts-file\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383291    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-systemd\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383311    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-tmp\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/182f9264-f98f-401d-aa1d-001f404fc6af-ovn-node-metrics-cert\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-multus\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqhg\" (UniqueName: \"kubernetes.io/projected/8936c93b-d868-46be-83e4-b6d224d4b11c-kube-api-access-dkqhg\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-conf\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-etc-tuned\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383447    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.383533 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-socket-dir-parent\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383494    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-etc-kubernetes\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-var-lib-kubelet\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-systemd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-agent-certs\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782ad8cb-0365-4cb1-86b0-4d3059401b91-tmp-dir\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383618    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-cnibin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383645    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-node-log\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-log-socket\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5jh\" (UniqueName: \"kubernetes.io/projected/683fd3fa-45a0-429a-9146-237713d74f5d-kube-api-access-rn5jh\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-systemd-units\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-daemon-config\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383828    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-multus-certs\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-root\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-metrics-client-ca\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383890    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3ec7504-ce41-4598-b6e6-1c8706119621-serviceca\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.384368 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-env-overrides\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.383979    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-system-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-konnectivity-ca\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-tls\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-sys\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-host\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384122    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-textfile\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384165    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-wtmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384170    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-sys\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384190    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-dbus\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-kubernetes\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384221    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-etc-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384245    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-slash\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-netns\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384304    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-ovn\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384329    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-bin\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385115 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-config\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-script-lib\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-kubelet\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3ec7504-ce41-4598-b6e6-1c8706119621-host\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6lw5\" (UniqueName: \"kubernetes.io/projected/e3ec7504-ce41-4598-b6e6-1c8706119621-kube-api-access-f6lw5\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384545    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-modprobe-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384569    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysconfig\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-kubelet\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-var-lib-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vmp\" (UniqueName: \"kubernetes.io/projected/6ac72c66-33a8-476a-afc7-093dc998aa4e-kube-api-access-46vmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384729    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384755    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-k8s-cni-cncf-io\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-netns\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-bin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384918    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-bin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.385939 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.384964    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-multus-certs\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385006    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-root\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385215    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-conf\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385281    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385329    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-lib-modules\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-multus\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385400    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-kubelet-config\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385455    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-run\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385483    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-cni-binary-copy\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385540    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-socket-dir-parent\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385656    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-netd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.385809    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385870    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-etc-kubernetes\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.385892    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs podName:eaf49b15-48f0-4abc-8f46-786db6035d5c nodeName:}" failed. No retries permitted until 2026-10-29 22:41:45.885871579 +0000 UTC m=+3.102908534 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs") pod "network-metrics-daemon-6nqg4" (UID: "eaf49b15-48f0-4abc-8f46-786db6035d5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385940    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-systemd\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-var-lib-kubelet\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386028    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-systemd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.386727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.385287    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-hostroot\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386102    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-cnibin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386202    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysconfig\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.386273    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386275    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.386350    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret podName:0fe38f83-f940-4e6a-8f45-0148745b3693 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:45.886332918 +0000 UTC m=+3.103369859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret") pod "global-pull-secret-syncer-cn2r6" (UID: "0fe38f83-f940-4e6a-8f45-0148745b3693") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386397    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-node-log\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386417    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-system-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386442    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-log-socket\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-kubelet\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386742    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-modprobe-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386781    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-systemd-units\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386844    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3ec7504-ce41-4598-b6e6-1c8706119621-host\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386952    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-konnectivity-ca\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.386987    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-env-overrides\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387050    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-sys\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387103    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-host\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387353    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-metrics-client-ca\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.387498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387365    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-textfile\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387400    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-kubelet\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387444    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-var-lib-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387470    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-dbus\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387508    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-kubernetes\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387529    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-wtmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387540    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-slash\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387564    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387597    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-netns\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387625    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-ovn\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.387650    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-bin\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.388095    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388350 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.388332    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782ad8cb-0365-4cb1-86b0-4d3059401b91-tmp-dir\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.388895 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.388443    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-k8s-cni-cncf-io\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.388895 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.388484    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-daemon-config\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.388999 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.388499    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-netns\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.388999 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.388842    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-config\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.388999 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.388908    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-script-lib\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.389466 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.389413    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.389762 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.389740    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-tls\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.390086 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.390061    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-etc-tuned\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.390847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.390804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-agent-certs\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.390847 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.390822    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-tmp\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.391057 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.391035    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/182f9264-f98f-401d-aa1d-001f404fc6af-ovn-node-metrics-cert\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.392316 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.392289    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.400469 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.400445    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjph9\" (UniqueName: \"kubernetes.io/projected/eaf49b15-48f0-4abc-8f46-786db6035d5c-kube-api-access-bjph9\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:45.400566 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.400478    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfpb\" (UniqueName: \"kubernetes.io/projected/782ad8cb-0365-4cb1-86b0-4d3059401b91-kube-api-access-lnfpb\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.403190 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.403169    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9qw\" (UniqueName: \"kubernetes.io/projected/182f9264-f98f-401d-aa1d-001f404fc6af-kube-api-access-2q9qw\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.405861 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.405842    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqhg\" (UniqueName: \"kubernetes.io/projected/8936c93b-d868-46be-83e4-b6d224d4b11c-kube-api-access-dkqhg\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.410069 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.410050    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:45.410176 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.410073    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:45.410176 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.410086    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2bfp8 for pod openshift-network-diagnostics/network-check-target-m7w8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:45.410176 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.410169    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8 podName:f0a8442d-746d-4365-a4f5-936b62940823 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:45.91012924 +0000 UTC m=+3.127166192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2bfp8" (UniqueName: "kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8") pod "network-check-target-m7w8j" (UID: "f0a8442d-746d-4365-a4f5-936b62940823") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:45.413293 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.413272    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5jh\" (UniqueName: \"kubernetes.io/projected/683fd3fa-45a0-429a-9146-237713d74f5d-kube-api-access-rn5jh\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.413603 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.413572    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vmp\" (UniqueName: \"kubernetes.io/projected/6ac72c66-33a8-476a-afc7-093dc998aa4e-kube-api-access-46vmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.413715 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.413636    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6lw5\" (UniqueName: \"kubernetes.io/projected/e3ec7504-ce41-4598-b6e6-1c8706119621-kube-api-access-f6lw5\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.479668 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.479629    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:41:45.490609 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.490483    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2abeb81_5999_493f_9474_8bf5d01b32df.slice/crio-0a5acddc90a7c0fa9b437f13b82224ecd826b902652a5546aabeb352952b1ab6 WatchSource:0}: Error finding container 0a5acddc90a7c0fa9b437f13b82224ecd826b902652a5546aabeb352952b1ab6: Status 404 returned error can't find the container with id 0a5acddc90a7c0fa9b437f13b82224ecd826b902652a5546aabeb352952b1ab6
Oct 29 22:41:45.492863 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.492843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:41:45.503111 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.503076    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0893ab11_598b_4072_893b_720af09698a7.slice/crio-a595aeb68da09e0e16cce10f78512c1733436de038bb691c8bed81f4d23c949b WatchSource:0}: Error finding container a595aeb68da09e0e16cce10f78512c1733436de038bb691c8bed81f4d23c949b: Status 404 returned error can't find the container with id a595aeb68da09e0e16cce10f78512c1733436de038bb691c8bed81f4d23c949b
Oct 29 22:41:45.512926 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.512904    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:41:45.513029 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.512996    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:45.517081 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.517057    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:41:45.525026 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.525001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:41:45.529181 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.529142    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:41:45.549864 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.549842    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zksnr"
Oct 29 22:41:45.550906 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.550830    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782ad8cb_0365_4cb1_86b0_4d3059401b91.slice/crio-89954462a1cfb3b467a77458afd3eac8bc3b5197f9db6cb8332c87dfe4d569c0 WatchSource:0}: Error finding container 89954462a1cfb3b467a77458afd3eac8bc3b5197f9db6cb8332c87dfe4d569c0: Status 404 returned error can't find the container with id 89954462a1cfb3b467a77458afd3eac8bc3b5197f9db6cb8332c87dfe4d569c0
Oct 29 22:41:45.551583 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.551560    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:41:45.554571 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.554543    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683fd3fa_45a0_429a_9146_237713d74f5d.slice/crio-2150c57c55979f17cbc328172f5f30fed21d9f43163b51d9737a6f1b6033a707 WatchSource:0}: Error finding container 2150c57c55979f17cbc328172f5f30fed21d9f43163b51d9737a6f1b6033a707: Status 404 returned error can't find the container with id 2150c57c55979f17cbc328172f5f30fed21d9f43163b51d9737a6f1b6033a707
Oct 29 22:41:45.555624 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.555605    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:41:45.556517 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.556492    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6637f1_cc56_4ad1_b1dd_d7ed5e3bb2e8.slice/crio-657bec506e881ef6f2c7dc939427d280840c7f02471ddc6c9c6a5e2b32fd83d8 WatchSource:0}: Error finding container 657bec506e881ef6f2c7dc939427d280840c7f02471ddc6c9c6a5e2b32fd83d8: Status 404 returned error can't find the container with id 657bec506e881ef6f2c7dc939427d280840c7f02471ddc6c9c6a5e2b32fd83d8
Oct 29 22:41:45.557647 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.557531    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a33c42_a915_4f9c_bd12_311c839696f5.slice/crio-a0916d61d034e0ff161bfbf65dd2513fca429fae0930919268a33162db66678e WatchSource:0}: Error finding container a0916d61d034e0ff161bfbf65dd2513fca429fae0930919268a33162db66678e: Status 404 returned error can't find the container with id a0916d61d034e0ff161bfbf65dd2513fca429fae0930919268a33162db66678e
Oct 29 22:41:45.558482 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.558467    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac72c66_33a8_476a_afc7_093dc998aa4e.slice/crio-cd691d7f618be2fe926331b5a40c3120184f382094dd92db39bcda88ad2d2e0d WatchSource:0}: Error finding container cd691d7f618be2fe926331b5a40c3120184f382094dd92db39bcda88ad2d2e0d: Status 404 returned error can't find the container with id cd691d7f618be2fe926331b5a40c3120184f382094dd92db39bcda88ad2d2e0d
Oct 29 22:41:45.572907 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:41:45.572885    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ec7504_ce41_4598_b6e6_1c8706119621.slice/crio-c36cc115320acef08c2513377b71e45ba956027a76b669934c6156e120b6e0da WatchSource:0}: Error finding container c36cc115320acef08c2513377b71e45ba956027a76b669934c6156e120b6e0da: Status 404 returned error can't find the container with id c36cc115320acef08c2513377b71e45ba956027a76b669934c6156e120b6e0da
Oct 29 22:41:45.890213 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.889837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:45.890213 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.889992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:45.890213 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.889998    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:45.890213 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.890077    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs podName:eaf49b15-48f0-4abc-8f46-786db6035d5c nodeName:}" failed. No retries permitted until 2026-10-29 22:41:46.890055877 +0000 UTC m=+4.107092831 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs") pod "network-metrics-daemon-6nqg4" (UID: "eaf49b15-48f0-4abc-8f46-786db6035d5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:45.890213 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.890130    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:45.890213 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.890193    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret podName:0fe38f83-f940-4e6a-8f45-0148745b3693 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:46.890176498 +0000 UTC m=+4.107213434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret") pod "global-pull-secret-syncer-cn2r6" (UID: "0fe38f83-f940-4e6a-8f45-0148745b3693") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:45.991997 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:45.991318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:45.991997 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.991533    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:45.991997 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.991555    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:45.991997 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.991568    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2bfp8 for pod openshift-network-diagnostics/network-check-target-m7w8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:45.991997 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:45.991623    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8 podName:f0a8442d-746d-4365-a4f5-936b62940823 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:46.991604892 +0000 UTC m=+4.208641833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bfp8" (UniqueName: "kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8") pod "network-check-target-m7w8j" (UID: "f0a8442d-746d-4365-a4f5-936b62940823") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:46.216659 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.216592    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:45 +0000 UTC" deadline="2028-03-24 00:49:51.592023141 +0000 UTC"
Oct 29 22:41:46.216659 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.216656    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12266h8m5.375371144s"
Oct 29 22:41:46.254208 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.254098    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jfhv" event={"ID":"6ac72c66-33a8-476a-afc7-093dc998aa4e","Type":"ContainerStarted","Data":"cd691d7f618be2fe926331b5a40c3120184f382094dd92db39bcda88ad2d2e0d"}
Oct 29 22:41:46.271032 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.270994    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ltpj" event={"ID":"683fd3fa-45a0-429a-9146-237713d74f5d","Type":"ContainerStarted","Data":"2150c57c55979f17cbc328172f5f30fed21d9f43163b51d9737a6f1b6033a707"}
Oct 29 22:41:46.278126 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.278092    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rhshl" event={"ID":"782ad8cb-0365-4cb1-86b0-4d3059401b91","Type":"ContainerStarted","Data":"89954462a1cfb3b467a77458afd3eac8bc3b5197f9db6cb8332c87dfe4d569c0"}
Oct 29 22:41:46.283582 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.283553    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal" event={"ID":"5a74b888ca6cfc8b230ed1966a77b369","Type":"ContainerStarted","Data":"fd428a0255091019f8efbb20b192446d6ec3ea61a9461b3e8fec68552b6ed156"}
Oct 29 22:41:46.306379 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.306345    2574 generic.go:356] "Generic (PLEG): container finished" podID="a7f8c8d4e5e3caee624106946868ca7a" containerID="49b9963d8b2c8c1be521159d6d6da1eebdab546ad08f7f7ed13d180c9f6d98b4" exitCode=0
Oct 29 22:41:46.306537 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.306458    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal" event={"ID":"a7f8c8d4e5e3caee624106946868ca7a","Type":"ContainerDied","Data":"49b9963d8b2c8c1be521159d6d6da1eebdab546ad08f7f7ed13d180c9f6d98b4"}
Oct 29 22:41:46.323448 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.323415    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"4ecf31fc9e6e709be10e6a6494f20477229d02b959734e7ebf0b2b3a67aa5f18"}
Oct 29 22:41:46.335431 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.335357    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zksnr" event={"ID":"8936c93b-d868-46be-83e4-b6d224d4b11c","Type":"ContainerStarted","Data":"35844342395beeb62fe5329e7b94effb2052d681fc06b3dd6ee825737025b951"}
Oct 29 22:41:46.352213 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.352129    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal" podStartSLOduration=2.352109993 podStartE2EDuration="2.352109993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:46.318911695 +0000 UTC m=+3.535948662" watchObservedRunningTime="2026-10-29 22:41:46.352109993 +0000 UTC m=+3.569146951"
Oct 29 22:41:46.357132 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.357099    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerStarted","Data":"a0916d61d034e0ff161bfbf65dd2513fca429fae0930919268a33162db66678e"}
Oct 29 22:41:46.372421 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.372388    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dwfrq" event={"ID":"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8","Type":"ContainerStarted","Data":"657bec506e881ef6f2c7dc939427d280840c7f02471ddc6c9c6a5e2b32fd83d8"}
Oct 29 22:41:46.394514 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.394123    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r" event={"ID":"0893ab11-598b-4072-893b-720af09698a7","Type":"ContainerStarted","Data":"a595aeb68da09e0e16cce10f78512c1733436de038bb691c8bed81f4d23c949b"}
Oct 29 22:41:46.413955 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.413825    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ltvq9" event={"ID":"d2abeb81-5999-493f-9474-8bf5d01b32df","Type":"ContainerStarted","Data":"0a5acddc90a7c0fa9b437f13b82224ecd826b902652a5546aabeb352952b1ab6"}
Oct 29 22:41:46.434323 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.434281    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxjxd" event={"ID":"e3ec7504-ce41-4598-b6e6-1c8706119621","Type":"ContainerStarted","Data":"c36cc115320acef08c2513377b71e45ba956027a76b669934c6156e120b6e0da"}
Oct 29 22:41:46.901313 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.900384    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:46.901313 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:46.900581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:46.901313 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:46.900743    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:46.901313 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:46.900830    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret podName:0fe38f83-f940-4e6a-8f45-0148745b3693 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:48.900810836 +0000 UTC m=+6.117847786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret") pod "global-pull-secret-syncer-cn2r6" (UID: "0fe38f83-f940-4e6a-8f45-0148745b3693") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:46.901313 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:46.901130    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:46.901313 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:46.901184    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs podName:eaf49b15-48f0-4abc-8f46-786db6035d5c nodeName:}" failed. No retries permitted until 2026-10-29 22:41:48.901169506 +0000 UTC m=+6.118206454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs") pod "network-metrics-daemon-6nqg4" (UID: "eaf49b15-48f0-4abc-8f46-786db6035d5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:47.002242 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.001603    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:47.002242 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:47.001790    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:47.002242 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:47.001815    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:47.002242 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:47.001828    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2bfp8 for pod openshift-network-diagnostics/network-check-target-m7w8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:47.002242 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:47.001887    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8 podName:f0a8442d-746d-4365-a4f5-936b62940823 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:49.001868573 +0000 UTC m=+6.218905552 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bfp8" (UniqueName: "kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8") pod "network-check-target-m7w8j" (UID: "f0a8442d-746d-4365-a4f5-936b62940823") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:47.217433 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.217315    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:45 +0000 UTC" deadline="2028-08-07 00:12:52.697537008 +0000 UTC"
Oct 29 22:41:47.217433 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.217355    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15529h31m5.480186159s"
Oct 29 22:41:47.239448 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.238879    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:47.239448 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:47.239006    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:47.239448 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.239234    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:47.239448 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:47.239316    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:47.239448 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.239344    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:47.239448 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:47.239416    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:47.459609 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.459499    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal" event={"ID":"a7f8c8d4e5e3caee624106946868ca7a","Type":"ContainerStarted","Data":"fe6889054d29cadbc2f606e9cb0c0db5f3a1bc4a0cf31e0e4d2207c3be877e05"}
Oct 29 22:41:47.492717 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:47.491721    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal" podStartSLOduration=3.491704988 podStartE2EDuration="3.491704988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:47.491001116 +0000 UTC m=+4.708038115" watchObservedRunningTime="2026-10-29 22:41:47.491704988 +0000 UTC m=+4.708741947"
Oct 29 22:41:48.488349 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:48.487575    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerStarted","Data":"421d536c8667ee94c3fecc189e914f6ad09fc1a359c8540b1b60f64a9833a715"}
Oct 29 22:41:48.492656 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:48.491565    2574 generic.go:356] "Generic (PLEG): container finished" podID="6ac72c66-33a8-476a-afc7-093dc998aa4e" containerID="4c36de1b23ae607076d48ad781ef960e7fec8eb023876e1789b84fa2db9c5bd8" exitCode=0
Oct 29 22:41:48.492656 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:48.492551    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jfhv" event={"ID":"6ac72c66-33a8-476a-afc7-093dc998aa4e","Type":"ContainerDied","Data":"4c36de1b23ae607076d48ad781ef960e7fec8eb023876e1789b84fa2db9c5bd8"}
Oct 29 22:41:48.916970 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:48.916881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:48.917120 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:48.917010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:48.917224 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:48.917142    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:48.917278 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:48.917227    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs podName:eaf49b15-48f0-4abc-8f46-786db6035d5c nodeName:}" failed. No retries permitted until 2026-10-29 22:41:52.917206923 +0000 UTC m=+10.134243861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs") pod "network-metrics-daemon-6nqg4" (UID: "eaf49b15-48f0-4abc-8f46-786db6035d5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:48.917557 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:48.917542    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:48.917621 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:48.917583    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret podName:0fe38f83-f940-4e6a-8f45-0148745b3693 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:52.917573979 +0000 UTC m=+10.134610914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret") pod "global-pull-secret-syncer-cn2r6" (UID: "0fe38f83-f940-4e6a-8f45-0148745b3693") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:49.018226 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.018081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:49.018403 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:49.018261    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:49.018403 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:49.018284    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:49.018403 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:49.018297    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2bfp8 for pod openshift-network-diagnostics/network-check-target-m7w8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:49.018403 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:49.018361    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8 podName:f0a8442d-746d-4365-a4f5-936b62940823 nodeName:}" failed. No retries permitted until 2026-10-29 22:41:53.018341074 +0000 UTC m=+10.235378032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bfp8" (UniqueName: "kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8") pod "network-check-target-m7w8j" (UID: "f0a8442d-746d-4365-a4f5-936b62940823") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:49.238369 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.238340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:49.238484 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:49.238466    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:49.238850 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.238830    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:49.238962 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:49.238941    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:49.239016 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.239007    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:49.239090 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:49.239073    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:49.501762 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.501688    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="421d536c8667ee94c3fecc189e914f6ad09fc1a359c8540b1b60f64a9833a715" exitCode=0
Oct 29 22:41:49.502221 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.501764    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerDied","Data":"421d536c8667ee94c3fecc189e914f6ad09fc1a359c8540b1b60f64a9833a715"}
Oct 29 22:41:49.517941 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.517440    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dwfrq" event={"ID":"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8","Type":"ContainerStarted","Data":"a2b21cfa3c700fd3a8c371b5c603c6e13edf15f56008f168469027d654983445"}
Oct 29 22:41:49.521087 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.520561    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jfhv" event={"ID":"6ac72c66-33a8-476a-afc7-093dc998aa4e","Type":"ContainerStarted","Data":"a94b8d1457c8faede32ddbc8d28d0b527bfcc678c4eb24bcad5bb73fd6719255"}
Oct 29 22:41:49.521087 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.520611    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6jfhv" event={"ID":"6ac72c66-33a8-476a-afc7-093dc998aa4e","Type":"ContainerStarted","Data":"ae6a407064a6865283e73c5f09ddcab0b5d0d278ff40d906e15882e486473eec"}
Oct 29 22:41:49.627606 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:49.625164    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dwfrq" podStartSLOduration=3.093173127 podStartE2EDuration="6.625137213s" totalImagesPullingTime="3.531964086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:49.584231405 +0000 UTC m=+6.801268431" watchObservedRunningTime="2026-10-29 22:41:49.625137213 +0000 UTC m=+6.842174202"
Oct 29 22:41:50.513186 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:50.512966    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:50.513645 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:50.513551    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:50.549900 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:50.549838    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6jfhv" podStartSLOduration=5.021291841 podStartE2EDuration="7.549818332s" totalImagesPullingTime="2.426532578s" totalInitContainerRuntime="101.993913ms" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:49.626033632 +0000 UTC m=+6.843070589" watchObservedRunningTime="2026-10-29 22:41:50.549818332 +0000 UTC m=+7.766855290"
Oct 29 22:41:51.245231 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:51.245191    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:51.245386 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:51.245332    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:51.245747 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:51.245719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:51.245876 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:51.245850    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:51.245924 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:51.245916    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:51.246013 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:51.245988    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:51.531550 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:51.531465    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r" event={"ID":"0893ab11-598b-4072-893b-720af09698a7","Type":"ContainerStarted","Data":"490fdde91da9c29dc115881bd5c8a5ef8d8649873a3c881dd3bd38a7185afa33"}
Oct 29 22:41:51.534921 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:51.534897    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:41:51.535801 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:51.535780    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qxjxd" event={"ID":"e3ec7504-ce41-4598-b6e6-1c8706119621","Type":"ContainerStarted","Data":"2d97200aecf1d0fe0ee12763122e30838795560589f6cddf8e48b87163133db8"}
Oct 29 22:41:52.909222 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:52.908718    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:41:52.957425 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:52.957390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:52.957566 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:52.957450    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:52.957621 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:52.957575    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:52.957682 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:52.957637    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret podName:0fe38f83-f940-4e6a-8f45-0148745b3693 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:00.957618862 +0000 UTC m=+18.174655820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret") pod "global-pull-secret-syncer-cn2r6" (UID: "0fe38f83-f940-4e6a-8f45-0148745b3693") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:41:52.957940 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:52.957923    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:52.958050 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:52.957965    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs podName:eaf49b15-48f0-4abc-8f46-786db6035d5c nodeName:}" failed. No retries permitted until 2026-10-29 22:42:00.957955564 +0000 UTC m=+18.174992512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs") pod "network-metrics-daemon-6nqg4" (UID: "eaf49b15-48f0-4abc-8f46-786db6035d5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:41:53.057850 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.057823    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:53.058025 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:53.058004    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:41:53.058097 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:53.058032    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:41:53.058097 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:53.058045    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2bfp8 for pod openshift-network-diagnostics/network-check-target-m7w8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:53.058425 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:53.058111    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8 podName:f0a8442d-746d-4365-a4f5-936b62940823 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:01.058090388 +0000 UTC m=+18.275127340 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bfp8" (UniqueName: "kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8") pod "network-check-target-m7w8j" (UID: "f0a8442d-746d-4365-a4f5-936b62940823") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:41:53.235648 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.235538    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:41:52.90874602Z","UUID":"b242c193-95f6-47e2-88c0-12c11c4822f1","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:41:53.245462 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.245425    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:53.245462 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.245462    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:53.245462 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.245437    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:53.245893 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.245876    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:41:53.245961 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.245905    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:41:53.247638 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:53.246691    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:53.247638 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:53.247171    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:53.247638 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:53.247247    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:53.544386 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.544347    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zksnr" event={"ID":"8936c93b-d868-46be-83e4-b6d224d4b11c","Type":"ContainerStarted","Data":"8d883a4f77c1cfce24cb12aea6e3caa4fa87f722974279e76faeaa0b09c2283a"}
Oct 29 22:41:53.549550 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.549519    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r" event={"ID":"0893ab11-598b-4072-893b-720af09698a7","Type":"ContainerStarted","Data":"14b1948d616888f0c92c84bf971e3e0b6247bdcb17eda926e919a10b06c7153d"}
Oct 29 22:41:53.580618 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.580521    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zksnr" podStartSLOduration=3.170701668 podStartE2EDuration="10.580501098s" totalImagesPullingTime="7.40979943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:53.579557356 +0000 UTC m=+10.796594314" watchObservedRunningTime="2026-10-29 22:41:53.580501098 +0000 UTC m=+10.797538056"
Oct 29 22:41:53.580759 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.580668    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qxjxd" podStartSLOduration=5.536456754 podStartE2EDuration="10.580662541s" totalImagesPullingTime="5.044205787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:51.568174581 +0000 UTC m=+8.785211539" watchObservedRunningTime="2026-10-29 22:41:53.580662541 +0000 UTC m=+10.797699498"
Oct 29 22:41:53.750898 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.750595    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:53.833224 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:53.833126    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:54.557883 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:54.557846    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="400e0b19707f14f3a1d1efdbbb58e41c3674db739f7bbc6555ff28f6d24bf9d2" exitCode=0
Oct 29 22:41:54.558474 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:54.557940    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerDied","Data":"400e0b19707f14f3a1d1efdbbb58e41c3674db739f7bbc6555ff28f6d24bf9d2"}
Oct 29 22:41:54.561602 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:54.561572    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r" event={"ID":"0893ab11-598b-4072-893b-720af09698a7","Type":"ContainerStarted","Data":"22b73b058d247bf9fc3833129198703e3fde50922389bf50a0edf33ae4a43d4a"}
Oct 29 22:41:54.564180 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:54.564137    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rhshl" event={"ID":"782ad8cb-0365-4cb1-86b0-4d3059401b91","Type":"ContainerStarted","Data":"f91615ceabfb88932f10014814cd46bc2a29e608fcb8e64feebed25e9878aa89"}
Oct 29 22:41:54.653786 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:54.653684    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r" podStartSLOduration=3.555748468 podStartE2EDuration="11.653669162s" totalImagesPullingTime="8.097920694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:54.653589418 +0000 UTC m=+11.870626376" watchObservedRunningTime="2026-10-29 22:41:54.653669162 +0000 UTC m=+11.870706121"
Oct 29 22:41:54.690681 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:54.690178    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rhshl" podStartSLOduration=3.908176809 podStartE2EDuration="11.690162134s" totalImagesPullingTime="7.781985325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:54.689479548 +0000 UTC m=+11.906516503" watchObservedRunningTime="2026-10-29 22:41:54.690162134 +0000 UTC m=+11.907199085"
Oct 29 22:41:55.243456 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:55.243425    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:55.243644 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:55.243560    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:55.243956 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:55.243933    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:55.244077 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:55.244029    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:55.244136 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:55.244083    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:55.244257 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:55.244168    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:55.567359 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:55.567275    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ltvq9" event={"ID":"d2abeb81-5999-493f-9474-8bf5d01b32df","Type":"ContainerStarted","Data":"583515a79809b3845c863d654e78c4bd68d7c597db29c62436c6a5347818f85c"}
Oct 29 22:41:55.600435 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:55.600381    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ltvq9" podStartSLOduration=4.443387213 podStartE2EDuration="12.600365499s" totalImagesPullingTime="8.156978286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:55.599899178 +0000 UTC m=+12.816936153" watchObservedRunningTime="2026-10-29 22:41:55.600365499 +0000 UTC m=+12.817402455"
Oct 29 22:41:56.512606 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:56.512567    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:56.512776 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:56.512730    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:41:56.513250 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:56.513222    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:41:56.572015 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:56.571980    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="1cd66bf33d7ff1952762ea0ebb34e24bc91e056673a45c00009cec998101625a" exitCode=0
Oct 29 22:41:56.572709 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:56.572080    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerDied","Data":"1cd66bf33d7ff1952762ea0ebb34e24bc91e056673a45c00009cec998101625a"}
Oct 29 22:41:56.574498 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:56.574471    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6ltpj" event={"ID":"683fd3fa-45a0-429a-9146-237713d74f5d","Type":"ContainerStarted","Data":"148257986a2b14b35d3a3ee21113a2d7be311e2c6578acdcfbfc9274ff4e88e2"}
Oct 29 22:41:56.601864 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:56.601827    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:57.241791 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:57.241762    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:57.241949 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:57.241823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:57.241949 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:57.241941    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:57.242057 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:57.241957    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:57.242057 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:57.242038    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:57.242160 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:57.242131    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:57.445757 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:57.445708    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:41:57.579129 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:57.579029    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="c5d87a3325225a66f57bd7b955426be267caaad133f2d9b85c5ab38da381bd8b" exitCode=0
Oct 29 22:41:57.579631 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:57.579612    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerDied","Data":"c5d87a3325225a66f57bd7b955426be267caaad133f2d9b85c5ab38da381bd8b"}
Oct 29 22:41:57.617676 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:57.617618    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6ltpj" podStartSLOduration=4.218863125 podStartE2EDuration="14.617599237s" totalImagesPullingTime="10.398736112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:41:56.642474583 +0000 UTC m=+13.859511539" watchObservedRunningTime="2026-10-29 22:41:57.617599237 +0000 UTC m=+14.834636192"
Oct 29 22:41:59.238437 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:59.238399    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:41:59.238437 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:59.238436    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:41:59.239031 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:59.238400    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:41:59.239031 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:59.238515    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:41:59.239031 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:59.238595    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:41:59.239031 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:41:59.238667    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:41:59.586356 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:41:59.586324    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerStarted","Data":"cae576831a381f3ee880c045c2c49e8e65cee30ccc025d1404652aa7d68ff8ab"}
Oct 29 22:42:00.591410 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.591169    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"a5262c5c4ef456d57d846a7a3636118a380dc1adf3dc007245b642a33e161c59"}
Oct 29 22:42:00.591835 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.591431    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"f775668901ebd8590394398a3312591e40c73008912efe3d8d88fe0d08323a3a"}
Oct 29 22:42:00.591835 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.591471    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"626dbc8abb778e279f8e57d3aac02bd40437dc0ff30374227cf0d26cc0e434ee"}
Oct 29 22:42:00.591835 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.591506    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"da26b6abffdae365f8e7779174f4a1b9711661cce143351247a33e78d79aac88"}
Oct 29 22:42:00.591835 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.591533    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"16a2264f1427c07976e963ca98476ec69b0defcac71434ecff228d14f05a0aa1"}
Oct 29 22:42:00.591835 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.591558    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"61e6b1c7fd7721cf7c910e04fd937f753700df0c6cf30a7c169435e1976a8123"}
Oct 29 22:42:00.593870 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.593841    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="cae576831a381f3ee880c045c2c49e8e65cee30ccc025d1404652aa7d68ff8ab" exitCode=0
Oct 29 22:42:00.593987 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:00.593890    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerDied","Data":"cae576831a381f3ee880c045c2c49e8e65cee30ccc025d1404652aa7d68ff8ab"}
Oct 29 22:42:01.030180 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.030077    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:01.030180 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.030128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:01.030350 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.030245    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:01.030350 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.030302    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret podName:0fe38f83-f940-4e6a-8f45-0148745b3693 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.030288969 +0000 UTC m=+34.247325917 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret") pod "global-pull-secret-syncer-cn2r6" (UID: "0fe38f83-f940-4e6a-8f45-0148745b3693") : object "kube-system"/"original-pull-secret" not registered
Oct 29 22:42:01.030350 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.030245    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:01.030446 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.030376    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs podName:eaf49b15-48f0-4abc-8f46-786db6035d5c nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.030365429 +0000 UTC m=+34.247402379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs") pod "network-metrics-daemon-6nqg4" (UID: "eaf49b15-48f0-4abc-8f46-786db6035d5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 29 22:42:01.130453 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.130411    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:01.130602 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.130575    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 29 22:42:01.130602 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.130594    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 29 22:42:01.130602 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.130603    2574 projected.go:196] Error preparing data for projected volume kube-api-access-2bfp8 for pod openshift-network-diagnostics/network-check-target-m7w8j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:01.130730 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.130653    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8 podName:f0a8442d-746d-4365-a4f5-936b62940823 nodeName:}" failed. No retries permitted until 2026-10-29 22:42:17.130636719 +0000 UTC m=+34.347673667 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bfp8" (UniqueName: "kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8") pod "network-check-target-m7w8j" (UID: "f0a8442d-746d-4365-a4f5-936b62940823") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 29 22:42:01.241632 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.241602    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:01.241632 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.241622    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:01.241792 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.241604    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:01.241792 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.241702    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:42:01.241792 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.241770    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:42:01.241905 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:01.241846    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:42:01.598682 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.598643    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="09c07ce8a3736f2f0a85d7010f9b16eed909dd6c65fcbcf12b71a9172eba5647" exitCode=0
Oct 29 22:42:01.599222 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:01.598684    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerDied","Data":"09c07ce8a3736f2f0a85d7010f9b16eed909dd6c65fcbcf12b71a9172eba5647"}
Oct 29 22:42:02.604001 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:02.603915    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"d3f23de07ba1357a7767f282908be2bb3c9b7ef2c81c544d9e08cd09ad655d60"}
Oct 29 22:42:02.607040 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:02.607013    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" event={"ID":"f7a33c42-a915-4f9c-bd12-311c839696f5","Type":"ContainerStarted","Data":"f0b57f4aae2dd336f166780b0fad87f4f98cdd6bfad7d1acfb52495b6a7bcc2f"}
Oct 29 22:42:02.664728 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:02.664678    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w2wcc" podStartSLOduration=8.873829635 podStartE2EDuration="19.664665699s" totalImagesPullingTime="10.790836064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:02.664421554 +0000 UTC m=+19.881458512" watchObservedRunningTime="2026-10-29 22:42:02.664665699 +0000 UTC m=+19.881702655"
Oct 29 22:42:03.239333 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:03.239299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:03.239496 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:03.239389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:03.239496 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:03.239385    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:42:03.239596 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:03.239551    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:42:03.239596 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:03.239590    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:03.239730 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:03.239710    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:42:04.615941 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:04.615744    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" event={"ID":"182f9264-f98f-401d-aa1d-001f404fc6af","Type":"ContainerStarted","Data":"71a770f2639f5b19fc538fdcc6030a8fa27b30561815ec829caf45e5dbbe8b48"}
Oct 29 22:42:04.616430 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:04.616026    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:04.631231 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:04.631177    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:04.689400 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:04.689343    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst" podStartSLOduration=7.319317742 podStartE2EDuration="21.689331752s" totalImagesPullingTime="14.37001401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:04.689099383 +0000 UTC m=+21.906136342" watchObservedRunningTime="2026-10-29 22:42:04.689331752 +0000 UTC m=+21.906368709"
Oct 29 22:42:05.238757 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:05.238726    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:05.238757 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:05.238742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:05.238945 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:05.238859    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:42:05.238945 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:05.238909    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:05.239045 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:05.239023    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:42:05.239111 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:05.239098    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:42:05.621802 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:05.619073    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:05.622379 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:05.619371    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:05.640988 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:05.640960    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:06.493300 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:06.493224    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6nqg4"]
Oct 29 22:42:06.493427 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:06.493373    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:06.493493 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:06.493474    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:42:06.495882 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:06.495864    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m7w8j"]
Oct 29 22:42:06.496016 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:06.496003    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:06.496116 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:06.496098    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:42:06.496593 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:06.496565    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cn2r6"]
Oct 29 22:42:06.496689 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:06.496662    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:06.496803 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:06.496736    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:42:06.620780 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:06.620752    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:07.623221 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:07.623194    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:08.239182 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:08.239131    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:08.239182 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:08.239142    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:08.239413 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:08.239142    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:08.239413 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:08.239263    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:42:08.239413 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:08.239352    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:42:08.239514 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:08.239444    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:42:10.238945 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:10.238914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:10.238945 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:10.238940    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:10.239378 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:10.238915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:10.239378 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:10.239008    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:42:10.239378 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:10.239096    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:42:10.239378 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:10.239162    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:42:12.238531 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.238493    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:12.238531 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.238523    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:12.238963 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.238593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:12.238963 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:12.238608    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6nqg4" podUID="eaf49b15-48f0-4abc-8f46-786db6035d5c"
Oct 29 22:42:12.238963 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:12.238676    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m7w8j" podUID="f0a8442d-746d-4365-a4f5-936b62940823"
Oct 29 22:42:12.238963 ip-10-0-136-187 kubenswrapper[2574]: E1029 22:42:12.238718    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cn2r6" podUID="0fe38f83-f940-4e6a-8f45-0148745b3693"
Oct 29 22:42:12.588503 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.588417    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeReady"
Oct 29 22:42:12.588664 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.588570    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 29 22:42:12.649895 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.649857    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-692j2"]
Oct 29 22:42:12.653076 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.652910    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kf7bp"]
Oct 29 22:42:12.653172 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.653020    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.656609 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.656589    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.665387 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.665367    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:42:12.665387 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.665375    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vstz\""
Oct 29 22:42:12.665511 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.665375    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:42:12.670357 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.670340    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pxqn\""
Oct 29 22:42:12.675475 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.675454    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-692j2"]
Oct 29 22:42:12.694407 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.694381    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kf7bp"]
Oct 29 22:42:12.746495 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.746464    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k7l64"]
Oct 29 22:42:12.753099 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.753075    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:12.765320 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.765292    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:42:12.765320 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.765301    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:42:12.765485 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.765292    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:42:12.765485 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.765334    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:42:12.767300 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.767283    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:42:12.768714 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.768696    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k7l64"]
Oct 29 22:42:12.820081 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820046    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b813c28-6b1a-429b-a812-5fcffab51b0c-crio-socket\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.820241 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820175    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b813c28-6b1a-429b-a812-5fcffab51b0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.820241 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d13647-137d-4aff-a28b-705df3a5b728-metrics-tls\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.820241 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820233    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b813c28-6b1a-429b-a812-5fcffab51b0c-data-volume\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.820378 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820295    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.820412 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqnvl\" (UniqueName: \"kubernetes.io/projected/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-api-access-nqnvl\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.820450 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t857q\" (UniqueName: \"kubernetes.io/projected/49d13647-137d-4aff-a28b-705df3a5b728-kube-api-access-t857q\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.820497 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d13647-137d-4aff-a28b-705df3a5b728-config-volume\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.820497 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.820479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d13647-137d-4aff-a28b-705df3a5b728-tmp-dir\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.865508 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.865439    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:42:12.865508 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.865460    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6bbfq\""
Oct 29 22:42:12.865508 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.865475    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:42:12.920828 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.920791    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d13647-137d-4aff-a28b-705df3a5b728-config-volume\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.920828 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.920829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d13647-137d-4aff-a28b-705df3a5b728-tmp-dir\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.921035 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.920853    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1185ece1-18e8-493f-a18c-b367c9793c32-cert\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:12.921035 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.920885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b813c28-6b1a-429b-a812-5fcffab51b0c-crio-socket\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.921035 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.920931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b813c28-6b1a-429b-a812-5fcffab51b0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.921035 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.920993    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d13647-137d-4aff-a28b-705df3a5b728-metrics-tls\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.921255 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b813c28-6b1a-429b-a812-5fcffab51b0c-data-volume\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.921255 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.921255 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921115    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqnvl\" (UniqueName: \"kubernetes.io/projected/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-api-access-nqnvl\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.921255 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t857q\" (UniqueName: \"kubernetes.io/projected/49d13647-137d-4aff-a28b-705df3a5b728-kube-api-access-t857q\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.921255 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921140    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b813c28-6b1a-429b-a812-5fcffab51b0c-crio-socket\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.921255 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nggt\" (UniqueName: \"kubernetes.io/projected/1185ece1-18e8-493f-a18c-b367c9793c32-kube-api-access-7nggt\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:12.921645 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921615    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b813c28-6b1a-429b-a812-5fcffab51b0c-data-volume\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.921738 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921649    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d13647-137d-4aff-a28b-705df3a5b728-tmp-dir\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.921794 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.921765    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d13647-137d-4aff-a28b-705df3a5b728-config-volume\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.922901 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.922876    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.925062 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.925042    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b813c28-6b1a-429b-a812-5fcffab51b0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.925126 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.925063    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d13647-137d-4aff-a28b-705df3a5b728-metrics-tls\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.938249 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.938225    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t857q\" (UniqueName: \"kubernetes.io/projected/49d13647-137d-4aff-a28b-705df3a5b728-kube-api-access-t857q\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.938354 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.938335    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqnvl\" (UniqueName: \"kubernetes.io/projected/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-api-access-nqnvl\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:12.968291 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.968258    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:12.975701 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:12.975683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:13.021530 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.021502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nggt\" (UniqueName: \"kubernetes.io/projected/1185ece1-18e8-493f-a18c-b367c9793c32-kube-api-access-7nggt\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:13.021679 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.021547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1185ece1-18e8-493f-a18c-b367c9793c32-cert\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:13.024160 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.024114    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1185ece1-18e8-493f-a18c-b367c9793c32-cert\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:13.040318 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.040290    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nggt\" (UniqueName: \"kubernetes.io/projected/1185ece1-18e8-493f-a18c-b367c9793c32-kube-api-access-7nggt\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:13.064374 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.064340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:13.212786 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.212703    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:13.301217 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.301189    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-692j2"]
Oct 29 22:42:13.302171 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.302137    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kf7bp"]
Oct 29 22:42:13.306227 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:42:13.306200    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d13647_137d_4aff_a28b_705df3a5b728.slice/crio-a5da56c11df534005a197157c1fc916d1247099f25e2097e71fc259848745575 WatchSource:0}: Error finding container a5da56c11df534005a197157c1fc916d1247099f25e2097e71fc259848745575: Status 404 returned error can't find the container with id a5da56c11df534005a197157c1fc916d1247099f25e2097e71fc259848745575
Oct 29 22:42:13.306580 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:42:13.306492    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b813c28_6b1a_429b_a812_5fcffab51b0c.slice/crio-0a768d4be81fb4b468cb8900e30700de9f981931893f6cc3338c6fbf3527677f WatchSource:0}: Error finding container 0a768d4be81fb4b468cb8900e30700de9f981931893f6cc3338c6fbf3527677f: Status 404 returned error can't find the container with id 0a768d4be81fb4b468cb8900e30700de9f981931893f6cc3338c6fbf3527677f
Oct 29 22:42:13.354178 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.354128    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k7l64"]
Oct 29 22:42:13.357681 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:42:13.357535    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1185ece1_18e8_493f_a18c_b367c9793c32.slice/crio-d6516779127588e80842ff5c7f9a9c18f2c0e2f64b734a2ef099aa6a027a65e2 WatchSource:0}: Error finding container d6516779127588e80842ff5c7f9a9c18f2c0e2f64b734a2ef099aa6a027a65e2: Status 404 returned error can't find the container with id d6516779127588e80842ff5c7f9a9c18f2c0e2f64b734a2ef099aa6a027a65e2
Oct 29 22:42:13.639435 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.639336    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf7bp" event={"ID":"3b813c28-6b1a-429b-a812-5fcffab51b0c","Type":"ContainerStarted","Data":"9c33fd58520bb6e1ad5a0f8d9b910c7d27d7dc8aa0e7918292ff0b12399ba530"}
Oct 29 22:42:13.639435 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.639397    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf7bp" event={"ID":"3b813c28-6b1a-429b-a812-5fcffab51b0c","Type":"ContainerStarted","Data":"0a768d4be81fb4b468cb8900e30700de9f981931893f6cc3338c6fbf3527677f"}
Oct 29 22:42:13.640520 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.640496    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k7l64" event={"ID":"1185ece1-18e8-493f-a18c-b367c9793c32","Type":"ContainerStarted","Data":"d6516779127588e80842ff5c7f9a9c18f2c0e2f64b734a2ef099aa6a027a65e2"}
Oct 29 22:42:13.641537 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:13.641513    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-692j2" event={"ID":"49d13647-137d-4aff-a28b-705df3a5b728","Type":"ContainerStarted","Data":"a5da56c11df534005a197157c1fc916d1247099f25e2097e71fc259848745575"}
Oct 29 22:42:14.238923 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.238885    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:14.239301 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.239286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:14.240269 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.239641    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:14.251539 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.251507    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:42:14.251636 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.251624    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:42:14.261344 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.261243    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4f7f9\""
Oct 29 22:42:14.261344 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.261290    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:42:14.261344 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.261246    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:42:14.261344 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.261317    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p7zhx\""
Oct 29 22:42:14.261513 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.261378    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7wdgt\""
Oct 29 22:42:14.520082 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.519954    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:14.600459 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.600418    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:14.646489 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.646454    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf7bp" event={"ID":"3b813c28-6b1a-429b-a812-5fcffab51b0c","Type":"ContainerStarted","Data":"01a72116185f74a2e28a6ac0cd0ca5a54fd8cac386703c62d5349415d70f7bdf"}
Oct 29 22:42:14.972701 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.972619    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:14.972897 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.972878    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:14.996459 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:14.996430    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:15.652695 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:15.652618    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-692j2" event={"ID":"49d13647-137d-4aff-a28b-705df3a5b728","Type":"ContainerStarted","Data":"fb37a0b24c9b49dbcb4116d690bb63f3023fe730bdaf4eed69d3eb05ef718ff7"}
Oct 29 22:42:15.652695 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:15.652685    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-692j2" event={"ID":"49d13647-137d-4aff-a28b-705df3a5b728","Type":"ContainerStarted","Data":"cdc58cd406cdc1cd2413e3e23e73021cb24a614f4299ff9ec23c9529a1e055e6"}
Oct 29 22:42:15.687367 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:15.687051    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-692j2" podStartSLOduration=1.744707493 podStartE2EDuration="3.687036366s" totalImagesPullingTime="1.942328873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:15.686287582 +0000 UTC m=+32.903324564" watchObservedRunningTime="2026-10-29 22:42:15.687036366 +0000 UTC m=+32.904073323"
Oct 29 22:42:16.657809 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:16.657769    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k7l64" event={"ID":"1185ece1-18e8-493f-a18c-b367c9793c32","Type":"ContainerStarted","Data":"92f64c9614f50938aebcdb5f130377a263e0561cb4ed2963de59856ed239e1cf"}
Oct 29 22:42:16.658315 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:16.657974    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:16.692888 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:16.692836    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k7l64" podStartSLOduration=1.649978531 podStartE2EDuration="4.692821356s" totalImagesPullingTime="3.042842825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:16.691674372 +0000 UTC m=+33.908711351" watchObservedRunningTime="2026-10-29 22:42:16.692821356 +0000 UTC m=+33.909858313"
Oct 29 22:42:17.057417 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.057326    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:17.057417 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.057382    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:17.060142 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.060117    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:17.060142 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.060138    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:17.158727 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.158690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:17.161024 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.161004    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:17.278662 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.278629    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:17.290783 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.290759    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:17.296929 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.296904    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:17.599507 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.599198    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6nqg4"]
Oct 29 22:42:17.601818 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:42:17.601794    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf49b15_48f0_4abc_8f46_786db6035d5c.slice/crio-7f6ea547fbd9fcef858b7d348b6c8d906a525492e8e41eac8977b91fcaacc480 WatchSource:0}: Error finding container 7f6ea547fbd9fcef858b7d348b6c8d906a525492e8e41eac8977b91fcaacc480: Status 404 returned error can't find the container with id 7f6ea547fbd9fcef858b7d348b6c8d906a525492e8e41eac8977b91fcaacc480
Oct 29 22:42:17.622940 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.622911    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cn2r6"]
Oct 29 22:42:17.625671 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:42:17.625648    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe38f83_f940_4e6a_8f45_0148745b3693.slice/crio-16fdd7f8146325f3689466a415cf4d59c8fdca49806c243300670d5da4b07120 WatchSource:0}: Error finding container 16fdd7f8146325f3689466a415cf4d59c8fdca49806c243300670d5da4b07120: Status 404 returned error can't find the container with id 16fdd7f8146325f3689466a415cf4d59c8fdca49806c243300670d5da4b07120
Oct 29 22:42:17.650569 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.650545    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m7w8j"]
Oct 29 22:42:17.653523 ip-10-0-136-187 kubenswrapper[2574]: W1029 22:42:17.653496    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a8442d_746d_4365_a4f5_936b62940823.slice/crio-a782cc97fcb81c1d45bfde7da9e3481c16d0885129c647fbc7fb99dd7f22de29 WatchSource:0}: Error finding container a782cc97fcb81c1d45bfde7da9e3481c16d0885129c647fbc7fb99dd7f22de29: Status 404 returned error can't find the container with id a782cc97fcb81c1d45bfde7da9e3481c16d0885129c647fbc7fb99dd7f22de29
Oct 29 22:42:17.661877 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.661849    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m7w8j" event={"ID":"f0a8442d-746d-4365-a4f5-936b62940823","Type":"ContainerStarted","Data":"a782cc97fcb81c1d45bfde7da9e3481c16d0885129c647fbc7fb99dd7f22de29"}
Oct 29 22:42:17.662933 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.662912    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cn2r6" event={"ID":"0fe38f83-f940-4e6a-8f45-0148745b3693","Type":"ContainerStarted","Data":"16fdd7f8146325f3689466a415cf4d59c8fdca49806c243300670d5da4b07120"}
Oct 29 22:42:17.664740 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.664717    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kf7bp" event={"ID":"3b813c28-6b1a-429b-a812-5fcffab51b0c","Type":"ContainerStarted","Data":"8ac544d30d41be0aed4f339779e6283ac9f94eafd20757072c9a2d7f1e147131"}
Oct 29 22:42:17.665818 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.665795    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6nqg4" event={"ID":"eaf49b15-48f0-4abc-8f46-786db6035d5c","Type":"ContainerStarted","Data":"7f6ea547fbd9fcef858b7d348b6c8d906a525492e8e41eac8977b91fcaacc480"}
Oct 29 22:42:17.700120 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:17.700072    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kf7bp" podStartSLOduration=2.568345047 podStartE2EDuration="5.700059132s" totalImagesPullingTime="3.131714085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:42:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:17.699567653 +0000 UTC m=+34.916604643" watchObservedRunningTime="2026-10-29 22:42:17.700059132 +0000 UTC m=+34.917096117"
Oct 29 22:42:19.676850 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:19.676769    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6nqg4" event={"ID":"eaf49b15-48f0-4abc-8f46-786db6035d5c","Type":"ContainerStarted","Data":"436e7d54927ebbe879fbfdc5345c3007b1c66b447b5280f224f08e98c4a16450"}
Oct 29 22:42:19.676850 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:19.676829    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6nqg4" event={"ID":"eaf49b15-48f0-4abc-8f46-786db6035d5c","Type":"ContainerStarted","Data":"5ae5cccf2347430255fddab7d687d1fb2fa9f91ac6b5a4748a725c8532450937"}
Oct 29 22:42:19.710324 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:19.710268    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6nqg4" podStartSLOduration=35.35580255 podStartE2EDuration="36.710249675s" totalImagesPullingTime="1.354447125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:19.709993573 +0000 UTC m=+36.927030531" watchObservedRunningTime="2026-10-29 22:42:19.710249675 +0000 UTC m=+36.927286639"
Oct 29 22:42:21.685379 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:21.685343    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m7w8j" event={"ID":"f0a8442d-746d-4365-a4f5-936b62940823","Type":"ContainerStarted","Data":"d32a06da3f498d3e320bb23a30adad0f114656fc5826f99a8db3ee6b82e6f159"}
Oct 29 22:42:21.685922 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:21.685478    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:21.718370 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:21.718316    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m7w8j" podStartSLOduration=35.274511612 podStartE2EDuration="38.718300888s" totalImagesPullingTime="3.443789276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:21.717595788 +0000 UTC m=+38.934632745" watchObservedRunningTime="2026-10-29 22:42:21.718300888 +0000 UTC m=+38.935337844"
Oct 29 22:42:22.690211 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:22.690176    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cn2r6" event={"ID":"0fe38f83-f940-4e6a-8f45-0148745b3693","Type":"ContainerStarted","Data":"723604ea8adb7ee65e227cccaeec0f5de502e4101978c1c236f058c5bd26c9ab"}
Oct 29 22:42:22.723432 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:22.723384    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cn2r6" podStartSLOduration=34.823193389 podStartE2EDuration="39.723372355s" totalImagesPullingTime="4.900178966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 22:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:42:22.723236307 +0000 UTC m=+39.940273264" watchObservedRunningTime="2026-10-29 22:42:22.723372355 +0000 UTC m=+39.940409361"
Oct 29 22:42:22.824978 ip-10-0-136-187 kubenswrapper[2574]: I1029 22:42:22.824840    2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:22.824996 ip-10-0-136-187 systemd[1]: Stopping Kubernetes Kubelet...
Oct 29 22:42:22.841628 ip-10-0-136-187 systemd[1]: kubelet.service: Deactivated successfully.
Oct 29 22:42:22.841791 ip-10-0-136-187 systemd[1]: Stopped Kubernetes Kubelet.
Oct 29 22:42:22.841962 ip-10-0-136-187 systemd[1]: kubelet.service: Consumed 2.538s CPU time.
Oct 29 22:42:22.851753 ip-10-0-136-187 systemd[1]: Starting Kubernetes Kubelet...
Oct 29 22:42:22.925904 ip-10-0-136-187 kubenswrapper[4839]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:22.925904 ip-10-0-136-187 kubenswrapper[4839]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 29 22:42:22.925904 ip-10-0-136-187 kubenswrapper[4839]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:22.925904 ip-10-0-136-187 kubenswrapper[4839]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 29 22:42:22.927991 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.927909    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.927991 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.927991    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.927991 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.927995    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.927998    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928002    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928005    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928007    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928010    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928013    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928016    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928019    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928022    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928025    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928027    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928035    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928038    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928041    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928044    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928046    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928049    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928052    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.928085 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928054    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928058    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928063    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928066    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928069    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928072    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928075    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928078    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928080    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928083    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928086    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928089    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928092    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928095    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928098    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928101    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928105    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928107    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928110    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928112    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.928580 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928115    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928118    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928120    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928123    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928126    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928128    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928131    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928133    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928136    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928139    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928141    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928159    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928163    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928167    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928173    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928176    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928179    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928182    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928184    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928188    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.929077 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928190    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928193    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928196    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928198    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928201    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928203    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928206    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928208    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928211    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928214    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928216    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928219    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928221    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928224    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928226    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928229    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928232    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928235    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928237    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.929610 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928240    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928242    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928245    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928247    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928250    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928253    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928256    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928259    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928261    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928264    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928266    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928269    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928271    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928274    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928277    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928279    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928282    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928285    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928287    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928290    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.930081 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928292    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928295    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928300    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928303    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928306    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.928309    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929055    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929063    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929066    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929070    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929072    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929075    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929078    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929081    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929084    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929087    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929090    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929093    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929096    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929099    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.930614 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929101    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929104    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929107    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929110    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929112    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929115    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929118    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929120    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929123    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929126    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929128    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929131    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929133    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929136    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929138    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929141    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929183    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929187    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929190    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929192    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.931161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929195    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929198    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929201    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929204    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929207    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929209    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929212    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929214    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929217    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929219    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929222    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929225    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929228    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929230    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929233    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929235    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929238    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929240    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929243    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.931672 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929248    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929252    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929255    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929258    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929262    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929265    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929268    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929271    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929275    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929278    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929280    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929283    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929286    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929289    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929292    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929295    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929300    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929304    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929307    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.932161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929310    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929314    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929317    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929320    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929322    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929325    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929328    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929331    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929333    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929336    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929339    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929341    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929344    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929346    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929349    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929352    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929354    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929357    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929359    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.932635 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929362    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929365    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929368    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929370    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929373    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929376    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929378    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929381    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929383    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929386    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929389    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929393    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929396    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929398    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929401    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929947    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929954    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929957    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929960    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.933095 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929963    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929966    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929968    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929971    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929973    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929976    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929979    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929982    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929984    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929987    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929989    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929992    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929995    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.929997    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930000    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930003    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930005    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930008    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930012    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.933592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930016    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930020    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930024    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930027    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930032    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930043    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930046    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930048    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930051    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930054    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930057    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930059    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930062    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930065    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930067    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930070    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930072    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930075    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930078    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930080    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.934233 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930082    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930085    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930088    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930091    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930094    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930096    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930099    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930101    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930104    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930106    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930109    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930111    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930114    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930116    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930119    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930122    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930126    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930128    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930131    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930134    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.934717 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930136    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930139    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930141    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930158    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930161    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930163    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930166    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930169    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930171    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930174    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930177    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930179    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930182    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930184    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930187    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930189    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930193    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930195    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930198    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.935226 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930200    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930203    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930206    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930208    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930211    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930214    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930216    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930220    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930222    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930227    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930230    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930233    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930236    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930239    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930241    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930243    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930246    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930248    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930251    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930253    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.935720 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930256    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930258    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930261    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.930264    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930360    4839 flags.go:64] FLAG: --address="0.0.0.0"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930367    4839 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930374    4839 flags.go:64] FLAG: --anonymous-auth="true"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930379    4839 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930383    4839 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930390    4839 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930395    4839 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930399    4839 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930402    4839 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930405    4839 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930409    4839 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930413    4839 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930416    4839 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930419    4839 flags.go:64] FLAG: --cgroup-root=""
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930422    4839 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930425    4839 flags.go:64] FLAG: --client-ca-file=""
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930428    4839 flags.go:64] FLAG: --cloud-provider="external"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930432    4839 flags.go:64] FLAG: --cluster-dns="[]"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930436    4839 flags.go:64] FLAG: --cluster-domain=""
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930439    4839 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 29 22:42:22.936227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930442    4839 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930446    4839 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930450    4839 flags.go:64] FLAG: --container-log-max-files="5"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930454    4839 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930457    4839 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930460    4839 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930464    4839 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930467    4839 flags.go:64] FLAG: --contention-profiling="false"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930470    4839 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930473    4839 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930476    4839 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930479    4839 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930484    4839 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930487    4839 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930490    4839 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930493    4839 flags.go:64] FLAG: --enable-load-reader="false"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930496    4839 flags.go:64] FLAG: --enable-server="true"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930500    4839 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930504    4839 flags.go:64] FLAG: --event-burst="100"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930508    4839 flags.go:64] FLAG: --event-qps="50"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930511    4839 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930514    4839 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930517    4839 flags.go:64] FLAG: --eviction-hard=""
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930521    4839 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930524    4839 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 29 22:42:22.936844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930527    4839 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930530    4839 flags.go:64] FLAG: --eviction-soft=""
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930533    4839 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930536    4839 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930540    4839 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930544    4839 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930547    4839 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930549    4839 flags.go:64] FLAG: --fail-swap-on="true"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930552    4839 flags.go:64] FLAG: --feature-gates=""
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930559    4839 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930563    4839 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930566    4839 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930569    4839 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930572    4839 flags.go:64] FLAG: --healthz-port="10248"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930575    4839 flags.go:64] FLAG: --help="false"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930578    4839 flags.go:64] FLAG: --hostname-override="ip-10-0-136-187.ec2.internal"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930582    4839 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930585    4839 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930588    4839 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930591    4839 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930594    4839 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930597    4839 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930600    4839 flags.go:64] FLAG: --image-service-endpoint=""
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930603    4839 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 29 22:42:22.937498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930607    4839 flags.go:64] FLAG: --kube-api-burst="100"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930610    4839 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930614    4839 flags.go:64] FLAG: --kube-api-qps="50"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930617    4839 flags.go:64] FLAG: --kube-reserved=""
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930620    4839 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930623    4839 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930626    4839 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930629    4839 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930633    4839 flags.go:64] FLAG: --lock-file=""
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930635    4839 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930638    4839 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930641    4839 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930648    4839 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930651    4839 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930654    4839 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930657    4839 flags.go:64] FLAG: --logging-format="text"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930660    4839 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930663    4839 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930667    4839 flags.go:64] FLAG: --manifest-url=""
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930670    4839 flags.go:64] FLAG: --manifest-url-header=""
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930674    4839 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930677    4839 flags.go:64] FLAG: --max-open-files="1000000"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930681    4839 flags.go:64] FLAG: --max-pods="110"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930684    4839 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930687    4839 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 29 22:42:22.938087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930691    4839 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930694    4839 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930696    4839 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930700    4839 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930703    4839 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930709    4839 flags.go:64] FLAG: --node-status-max-images="50"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930712    4839 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930716    4839 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930720    4839 flags.go:64] FLAG: --pod-cidr=""
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930722    4839 flags.go:64] FLAG: --pod-manifest-path=""
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930725    4839 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930728    4839 flags.go:64] FLAG: --pods-per-core="0"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930731    4839 flags.go:64] FLAG: --port="10250"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930734    4839 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930737    4839 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08079b5c878559f90"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930740    4839 flags.go:64] FLAG: --qos-reserved=""
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930743    4839 flags.go:64] FLAG: --read-only-port="10255"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930746    4839 flags.go:64] FLAG: --register-node="true"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930749    4839 flags.go:64] FLAG: --register-with-taints=""
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930755    4839 flags.go:64] FLAG: --registry-burst="10"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930758    4839 flags.go:64] FLAG: --registry-qps="5"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930761    4839 flags.go:64] FLAG: --reserved-cpus=""
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930764    4839 flags.go:64] FLAG: --reserved-memory=""
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930768    4839 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930771    4839 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930774    4839 flags.go:64] FLAG: --rotate-certificates="false"
Oct 29 22:42:22.938722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930777    4839 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930781    4839 flags.go:64] FLAG: --runonce="false"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930783    4839 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930786    4839 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930789    4839 flags.go:64] FLAG: --seccomp-default="false"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930792    4839 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930795    4839 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930798    4839 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930802    4839 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930805    4839 flags.go:64] FLAG: --storage-driver-password="root"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930808    4839 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930810    4839 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930813    4839 flags.go:64] FLAG: --storage-driver-user="root"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930818    4839 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930821    4839 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930824    4839 flags.go:64] FLAG: --system-cgroups=""
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930826    4839 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930832    4839 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930834    4839 flags.go:64] FLAG: --tls-cert-file=""
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930837    4839 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930841    4839 flags.go:64] FLAG: --tls-min-version=""
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930844    4839 flags.go:64] FLAG: --tls-private-key-file=""
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930847    4839 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930850    4839 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930853    4839 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930857    4839 flags.go:64] FLAG: --v="2"
Oct 29 22:42:22.939477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930862    4839 flags.go:64] FLAG: --version="false"
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930866    4839 flags.go:64] FLAG: --vmodule=""
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930869    4839 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.930873    4839 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931004    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931008    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931011    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931014    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931018    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931021    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931024    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931027    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931029    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931031    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931034    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931036    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931039    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931041    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931044    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931048    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.940575 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931050    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931053    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931056    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931058    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931061    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931064    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931066    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931069    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931071    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931074    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931076    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931080    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931083    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931085    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931088    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931090    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931107    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931111    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931115    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931118    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.941073 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931121    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931124    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931127    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931129    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931132    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931135    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931137    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931140    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931155    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931158    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931161    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931165    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931168    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931172    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931176    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931179    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931182    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931184    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931187    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931190    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.941597 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931192    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931195    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931198    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931202    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931204    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931207    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931210    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931213    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931215    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931218    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931221    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931223    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931226    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931229    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931232    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931234    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931237    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931239    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931242    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.942103 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931244    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931247    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931250    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931252    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931256    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931258    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931261    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931263    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931266    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931269    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931271    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931274    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931276    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931279    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931281    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931284    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931287    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931290    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931293    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931296    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.942690 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931298    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931301    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931305    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931309    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931312    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931315    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931318    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931321    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931324    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931326    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931329    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.931354    4839 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931390    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931395    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931398    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931401    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.943298 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931403    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931406    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931409    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931412    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931414    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931417    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931419    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931422    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931424    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931427    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931430    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931432    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931436    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931439    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931442    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931444    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931447    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931450    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931452    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931455    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.943727 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931457    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931460    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931462    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931465    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931468    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931471    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931474    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931476    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931479    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931481    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931484    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931486    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931489    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931492    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931494    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931497    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931499    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931502    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931504    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931507    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.944482 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931509    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931512    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931515    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931517    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931521    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931524    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931526    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931529    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931531    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931534    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931536    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931539    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931543    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931546    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931549    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931552    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931554    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931557    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931560    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931562    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.944986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931565    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931568    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931570    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931573    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931576    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931579    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931582    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931584    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931587    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931589    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931592    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931594    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931598    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931616    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931620    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931623    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931627    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931630    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931632    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.945618 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931635    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931638    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931640    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931643    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931646    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931648    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931651    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931653    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931656    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931659    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931661    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931664    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931667    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931670    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931672    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931675    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931678    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931681    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931683    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.946119 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931686    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931690    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931694    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.931697    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.935655    4839 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.935671    4839 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.935689    4839 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.935694    4839 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935778    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935783    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935787    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935790    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935793    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935796    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935798    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935801    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935804    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935807    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935809    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.946622 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935812    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935815    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935817    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935820    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935823    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935831    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935835    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935837    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935840    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935843    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935845    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935848    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935851    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935853    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935856    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935859    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935862    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935865    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935868    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935870    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.947090 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935873    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935876    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935878    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935881    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935883    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935886    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935889    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935893    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935898    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935901    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935904    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935907    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935910    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935913    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935915    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935918    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935921    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935924    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935933    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.947592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935936    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935939    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935942    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935944    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935948    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935950    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935953    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935955    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935958    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935961    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935963    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935966    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935968    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935971    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935974    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935977    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935979    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935981    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935984    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935987    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.948070 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935989    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935992    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935994    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935997    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.935999    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936002    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936005    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936007    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936010    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936012    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936015    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936017    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936025    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936028    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936031    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936034    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936036    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936039    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936041    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936044    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.948568 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936047    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936050    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936053    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936055    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936058    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936060    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936063    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936065    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936069    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936072    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936075    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936078    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936080    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936083    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936085    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936087    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.949098 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.936113    4839 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936330    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936336    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936339    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936343    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936346    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936348    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936352    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936354    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936368    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936372    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936375    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936377    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936380    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936383    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936385    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936389    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936391    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936394    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.949588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936396    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936399    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936401    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936404    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936406    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936409    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936412    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936414    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936416    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936419    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936422    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936424    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936428    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936432    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936436    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936440    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936443    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936445    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936448    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.950043 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936451    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936453    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936456    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936459    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936468    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936471    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936474    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936476    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936479    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936481    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936484    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936487    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936489    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936492    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936495    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936497    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936500    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936502    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936505    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936508    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.950570 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936510    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936513    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936515    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936518    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936520    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936523    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936525    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936527    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936530    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936532    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936535    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936537    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936540    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936542    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936545    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936547    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936553    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936562    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936564    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936567    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.951058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936570    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936573    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936575    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936578    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936580    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936583    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936586    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936588    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936591    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936593    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936596    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936599    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936601    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936603    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936606    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936609    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936611    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936614    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936617    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936619    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.951547 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936622    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936624    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936627    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936629    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936632    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936634    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936637    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936639    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936643    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.936667    4839 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936721    4839 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936725    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936728    4839 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936731    4839 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936734    4839 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936737    4839 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 29 22:42:22.952040 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936740    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936742    4839 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936745    4839 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936749    4839 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936753    4839 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936757    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936760    4839 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936763    4839 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936765    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936768    4839 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936771    4839 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936774    4839 feature_gate.go:386] unrecognized feature gate: Example2
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936776    4839 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936779    4839 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936781    4839 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936784    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936786    4839 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936789    4839 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936791    4839 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936794    4839 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 29 22:42:22.952442 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936796    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936799    4839 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936801    4839 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936804    4839 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936806    4839 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936810    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936814    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936822    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936825    4839 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936828    4839 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936831    4839 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936834    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936836    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936838    4839 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936841    4839 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936843    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936846    4839 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936849    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936851    4839 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936854    4839 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 29 22:42:22.952929 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936856    4839 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936859    4839 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936863    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936867    4839 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936870    4839 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936874    4839 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936878    4839 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936882    4839 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936885    4839 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936889    4839 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936893    4839 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936897    4839 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936901    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936905    4839 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936909    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936913    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936917    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936925    4839 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936928    4839 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 29 22:42:22.953432 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936930    4839 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936941    4839 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936944    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936946    4839 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936949    4839 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936952    4839 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936954    4839 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936957    4839 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936959    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936962    4839 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936964    4839 feature_gate.go:386] unrecognized feature gate: Example
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936967    4839 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936970    4839 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936972    4839 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936975    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936978    4839 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936980    4839 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936982    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936985    4839 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936988    4839 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 29 22:42:22.953887 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936990    4839 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936992    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936995    4839 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.936997    4839 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937000    4839 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937004    4839 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937009    4839 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937016    4839 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937021    4839 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937026    4839 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937033    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937038    4839 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937042    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937046    4839 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937064    4839 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937069    4839 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937073    4839 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937078    4839 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937082    4839 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 29 22:42:22.954395 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937086    4839 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:42:22.937090    4839 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.937207    4839 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.938524    4839 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.938592    4839 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.938859    4839 server.go:1014] "Starting client certificate rotation"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.939040    4839 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.939349    4839 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-28 22:36:43 +0000 UTC" deadline="2028-04-28 02:58:49.853568935 +0000 UTC"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.939939    4839 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13108h16m26.913634438s"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.940392    4839 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.941834    4839 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.943336    4839 log.go:25] "Validated CRI v1 runtime API"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.944985    4839 log.go:25] "Validated CRI v1 image API"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.945803    4839 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 29 22:42:22.954843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.947489    4839 fs.go:135] Filesystem UUIDs: map[24898a22-b72a-4538-b907-684b669d2735:/dev/nvme0n1p4 38af0018-b430-40f9-a22f-13157c698ca9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 29 22:42:22.955229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.947510    4839 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a5acddc90a7c0fa9b437f13b82224ecd826b902652a5546aabeb352952b1ab6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a5acddc90a7c0fa9b437f13b82224ecd826b902652a5546aabeb352952b1ab6/userdata/shm major:0 minor:71 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a768d4be81fb4b468cb8900e30700de9f981931893f6cc3338c6fbf3527677f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a768d4be81fb4b468cb8900e30700de9f981931893f6cc3338c6fbf3527677f/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16fdd7f8146325f3689466a415cf4d59c8fdca49806c243300670d5da4b07120/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16fdd7f8146325f3689466a415cf4d59c8fdca49806c243300670d5da4b07120/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35844342395beeb62fe5329e7b94effb2052d681fc06b3dd6ee825737025b951/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35844342395beeb62fe5329e7b94effb2052d681fc06b3dd6ee825737025b951/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4ecf31fc9e6e709be10e6a6494f20477229d02b959734e7ebf0b2b3a67aa5f18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4ecf31fc9e6e709be10e6a6494f20477229d02b959734e7ebf0b2b3a67aa5f18/userdata/shm major:0 minor:105 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/657bec506e881ef6f2c7dc939427d280840c7f02471ddc6c9c6a5e2b32fd83d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/657bec506e881ef6f2c7dc939427d280840c7f02471ddc6c9c6a5e2b32fd83d8/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f6ea547fbd9fcef858b7d348b6c8d906a525492e8e41eac8977b91fcaacc480/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f6ea547fbd9fcef858b7d348b6c8d906a525492e8e41eac8977b91fcaacc480/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89954462a1cfb3b467a77458afd3eac8bc3b5197f9db6cb8332c87dfe4d569c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89954462a1cfb3b467a77458afd3eac8bc3b5197f9db6cb8332c87dfe4d569c0/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0916d61d034e0ff161bfbf65dd2513fca429fae0930919268a33162db66678e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0916d61d034e0ff161bfbf65dd2513fca429fae0930919268a33162db66678e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a595aeb68da09e0e16cce10f78512c1733436de038bb691c8bed81f4d23c949b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a595aeb68da09e0e16cce10f78512c1733436de038bb691c8bed81f4d23c949b/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a5da56c11df534005a197157c1fc916d1247099f25e2097e71fc259848745575/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a5da56c11df534005a197157c1fc916d1247099f25e2097e71fc259848745575/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a782cc97fcb81c1d45bfde7da9e3481c16d0885129c647fbc7fb99dd7f22de29/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a782cc97fcb81c1d45bfde7da9e3481c16d0885129c647fbc7fb99dd7f22de29/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c36cc115320acef08c2513377b71e45ba956027a76b669934c6156e120b6e0da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c36cc115320acef08c2513377b71e45ba956027a76b669934c6156e120b6e0da/userdata/shm major:0 minor:101 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd691d7f618be2fe926331b5a40c3120184f382094dd92db39bcda88ad2d2e0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd691d7f618be2fe926331b5a40c3120184f382094dd92db39bcda88ad2d2e0d/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6516779127588e80842ff5c7f9a9c18f2c0e2f64b734a2ef099aa6a027a65e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6516779127588e80842ff5c7f9a9c18f2c0e2f64b734a2ef099aa6a027a65e2/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0893ab11-598b-4072-893b-720af09698a7/volumes/kubernetes.io~projected/kube-api-access-jrtcx:{mountpoint:/var/lib/kubelet/pods/0893ab11-598b-4072-893b-720af09698a7/volumes/kubernetes.io~projected/kube-api-access-jrtcx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fe38f83-f940-4e6a-8f45-0148745b3693/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/0fe38f83-f940-4e6a-8f45-0148745b3693/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1185ece1-18e8-493f-a18c-b367c9793c32/volumes/kubernetes.io~projected/kube-api-access-7nggt:{mountpoint:/var/lib/kubelet/pods/1185ece1-18e8-493f-a18c-b367c9793c32/volumes/kubernetes.io~projected/kube-api-access-7nggt major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1185ece1-18e8-493f-a18c-b367c9793c32/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/1185ece1-18e8-493f-a18c-b367c9793c32/volumes/kubernetes.io~secret/cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volumes/kubernetes.io~projected/kube-api-access-2q9qw:{mountpoint:/var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volumes/kubernetes.io~projected/kube-api-access-2q9qw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b813c28-6b1a-429b-a812-5fcffab51b0c/volumes/kubernetes.io~projected/kube-api-access-nqnvl:{mountpoint:/var/lib/kubelet/pods/3b813c28-6b1a-429b-a812-5fcffab51b0c/volumes/kubernetes.io~projected/kube-api-access-nqnvl major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b813c28-6b1a-429b-a812-5fcffab51b0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/3b813c28-6b1a-429b-a812-5fcffab51b0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49d13647-137d-4aff-a28b-705df3a5b728/volumes/kubernetes.io~projected/kube-api-access-t857q:{mountpoint:/var/lib/kubelet/pods/49d13647-137d-4aff-a28b-705df3a5b728/volumes/kubernetes.io~projected/kube-api-access-t857q major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49d13647-137d-4aff-a28b-705df3a5b728/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/49d13647-137d-4aff-a28b-705df3a5b728/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~projected/kube-api-access-rn5jh:{mountpoint:/var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~projected/kube-api-access-rn5jh major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~projected/kube-api-access-46vmp:{mountpoint:/var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~projected/kube-api-access-46vmp major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/782ad8cb-0365-4cb1-86b0-4d3059401b91/volumes/kubernetes.io~projected/kube-api-access-lnfpb:{mountpoint:/var/lib/kubelet/pods/782ad8cb-0365-4cb1-86b0-4d3059401b91/volumes/kubernetes.io~projected/kube-api-access-lnfpb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8936c93b-d868-46be-83e4-b6d224d4b11c/volumes/kubernetes.io~projected/kube-api-access-dkqhg:{mountpoint:/var/lib/kubelet/pods/8936c93b-d868-46be-83e4-b6d224d4b11c/volumes/kubernetes.io~projected/kube-api-access-dkqhg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/volumes/kubernetes.io~secret/agent-certs major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2abeb81-5999-493f-9474-8bf5d01b32df/volumes/kubernetes.io~projected/kube-api-access-8f2qc:{mountpoint:/var/lib/kubelet/pods/d2abeb81-5999-493f-9474-8bf5d01b32df/volumes/kubernetes.io~projected/kube-api-access-8f2qc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3ec7504-ce41-4598-b6e6-1c8706119621/volumes/kubernetes.io~projected/kube-api-access-f6lw5:{mountpoint:/var/lib/kubelet/pods/e3ec7504-ce41-4598-b6e6-1c8706119621/volumes/kubernetes.io~projected/kube-api-access-f6lw5 major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaf49b15-48f0-4abc-8f46-786db6035d5c/volumes/kubernetes.io~projected/kube-api-access-bjph9:{mountpoint:/var/lib/kubelet/pods/eaf49b15-48f0-4abc-8f46-786db6035d5c/volumes/kubernetes.io~projected/kube-api-access-bjph9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaf49b15-48f0-4abc-8f46-786db6035d5c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/eaf49b15-48f0-4abc-8f46-786db6035d5c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0a8442d-746d-4365-a4f5-936b62940823/volumes/kubernetes.io~projected/kube-api-access-2bfp8:{mountpoint:/var/lib/kubelet/pods/f0a8442d-746d-4365-a4f5-936b62940823/volumes/kubernetes.io~projected/kube-api-access-2bfp8 major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7a33c42-a915-4f9c-bd12-311c839696f5/volumes/kubernetes.io~projected/kube-api-access-n6s75:{mountpoint:/var/lib/kubelet/pods/f7a33c42-a915-4f9c-bd12-311c839696f5/volumes/kubernetes.io~projected/kube-api-access-n6s75 major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-103:{mountpoint:/var/lib/containers/storage/overlay/3dc3eff01b3dad3b8721551ff6233fef8a8a10dc9bb31948aee913753ec80180/merged major:0 minor:103 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/429fbaa24e14a080cc51e54cad80c6e33986e80b44338a5c38eb8cfe611f6094/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/8cfb405b50cb607c1beb5a3336d53214594afcdde1650021678e90588d949b41/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/af5c84dca1f48e3eceec30819e3a2ed8f63ca3e313ee14e30ca79c48de95f2be/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/93c8de32430e3dba0f81a5c98c5d5c77e88e5dbfab02304a6b5e6e468d7ec9d4/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-121:{mountpoint:/var/lib/containers/storage/overlay/c9e568f58df0943a5168787e5eead93249826677ac712d8608bf724ba9437660/merged major:0 minor:121 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/containers/storage/overlay/54d86c88436acdcb02053d24738975e26f1fcd816cdbdd6a6a9f7a24df1f9b11/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/b1f9f7fc32010888516ef03ede78c70899362f13cb583dfdd6c3e011f7aeeff5/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/143d4836056f1d4eb48e0a60518e01bb2325163ae0e98ecb9d16afa67dce6efa/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/37024e5a6a7f7908e8aa0f7ddcc5774bb09f0fc50f584edb5bea2a29304be7dd/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e6917a0f107c5fe4dfca5b950aa2d7a67b4bda6fc42b7f2fbe8402bff06b4bdb/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/1d21c39617a4af51042cb101b5720e1145620d5a2d942871cc5e04f601f26aa3/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/f2761f3ec1855e2b9df2932ffc34ae1be095842963d866b5dcb014bd31fc274d/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/e4ff437058f1e5071d12f38e410a959c3387c5d2930f37370fc6c1d9d18f44ef/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/87d87331668b9d55cada65fe1b0d5173a29fcd43c95a36c66355a021d3072996/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/d43adf90aa876b04941adfa048ffe7824de4a17e6e82eb672b7c7bbf9a2ee218/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/6b2a7d5be758e970d623efdb20a55de645cc085e2413bbc20e8b3b43aa66616c/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/b18c1d448db84d8f86286b1b3eac1908ab5d4649a8ac47324c9be625e7e9d9da/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/ca9bc232147f788b49c541bb05351236ddd905cb8bc09fa74ddb98d7f0a44014/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/31b3071f4f2908fa6c04b4a2a5e48655ee51b70027ff54ff363d83844c9e2abb/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/652726e3cf77f49ff5429e961b68f252e846a66b7849f88308caf6b62612e750/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-199:{mountpoint:/var/lib/containers/storage/overlay/72b4aaef5d49fc653c6c52f5cbb1d8a2d4c01eccc62bd20e1f0170cfb825e9d2/merged major:0 minor:199 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/1f0545fadc3303a2e6067b2fea7d53eba1fb2dc0912c79d35bd388117ed15a90/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/3a1896e7fe5e9529aaafde9b318328eda787040b93afc486f7c6a3994e687d81/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/784d9ae7ac97c56b25a6760726c2a12f30b976e3c11c3a49213f3b22979505aa/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/4650b4d46d889d9ababe9259975fcd2ad6bae84fa7398513d6fab24fdcb595d4/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/c76b363aad378f9ef0dc83c196b96271887ad41ca9524f46700def1f6e8b125a/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/b13972ab07d06e8914a9b60dda240296ac6c300da8b2277ac687963b96477288/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/8c378be7f496e61efac7e13f5809c7c8396fad2ea426672405ca42442cfa76b6/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/29e82f40b1cc75adb7357f09a71a14b8123eeb12cbcd4b93d90de489afdc29a1/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/1fb11984fac2afce81229f2c3df3cbccea153a46d491d89a3d1c2d07fdd1b249/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/03a9214fd0126d84f9e305d1e1c2cbe56d4945b28ff3b3b8a2582064a1570b92/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/2f6f453b8b5f5a199b7eb99331ef3aa4d19273bb2fa1f418f80fee5aca946fd7/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/9dcf69f57f3f2d3f2b192f0def93606ba0dd010a06d63167a2cad1f3b1511cd3/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/4112a0c3e97aa17dfa49d13d8974627a5cb4d7377dc14532b304cd577dcde47b/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/e1f589e4bdc01898a93ce55a1b48aa6c943cfdfbf68fa56740c062b5fcd2145f/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/459a4b5a583b7da52a48bcf92cd17ab88ebdafae4521f5ec0d2a865502222cb9/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/de3bd9f44368b026ca12cebae9e404d6858fa759b965e9f36a64b22fd4587bee/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/5c40e880967b13af0425a290ac69d5da1a5ef87296b7cdc0fbee8cbb57bab8a2/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/34684e292e231a7f4b9efefa4cfd1e262cd70ba11d5368f0f84847ddc793b8ae/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/15467f5fc63c6697aee91efb7b6dc98411fdbc3e8a0d08c39d2b31860f88f506/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0045439be970d2ecd4b2f7bfc003a644405af4fe9a753b88bec026873b0336ed/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/4b6c50c1d1b38d6690f023bdd55ec337545b160171adc8e5577500b3dd9c4ba1/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/b777982fd5f9cf12eb2b9a48bbedf381aae2ff66e6e46a4b6977649603076418/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/81137e9c7dddeec5ba65d9a42ad99ab30f7a57cd3c73ad4ecf410b9a53648b1b/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/46f69be2830a1f3823c67a19404952610b8bf898cbc6298b9f0a8b468b05cdeb/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/3545e6142de22b3caf004a56c093c02b1223bcfcb56788ec217fe08183b2ae4f/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/9f641f893bcf73eacf44b951fbd4d36697901b06c9ccd55fdf6e43c4bc88e288/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/45aafc7501e157c8824d5c897d0b2f20c2ff0444f947009aa202061681d79fcb/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-99:{mountpoint:/var/lib/containers/storage/overlay/b73e89a3f37c742cd150eacb1cfb969d7350216d1ef2289475ef86d88d9bda36/merged major:0 minor:99 fsType:overlay blockSize:0}]
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954113    4839 manager.go:217] Machine: {Timestamp:2026-10-29 22:42:22.95357141 +0000 UTC m=+0.056647698 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083078 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ba95584ca1a013eec7ea328c58ce9 SystemUUID:ec2ba955-84ca-1a01-3eec-7ea328c58ce9 BootID:f6dd021f-7c1f-47f5-af62-32f447ef1529 Filesystems:[{Device:/var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a5acddc90a7c0fa9b437f13b82224ecd826b902652a5546aabeb352952b1ab6/userdata/shm DeviceMajor:0 DeviceMinor:71 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f6ea547fbd9fcef858b7d348b6c8d906a525492e8e41eac8977b91fcaacc480/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a782cc97fcb81c1d45bfde7da9e3481c16d0885129c647fbc7fb99dd7f22de29/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/eaf49b15-48f0-4abc-8f46-786db6035d5c/volumes/kubernetes.io~projected/kube-api-access-bjph9 DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c36cc115320acef08c2513377b71e45ba956027a76b669934c6156e120b6e0da/userdata/shm DeviceMajor:0 DeviceMinor:101 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6516779127588e80842ff5c7f9a9c18f2c0e2f64b734a2ef099aa6a027a65e2/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49d13647-137d-4aff-a28b-705df3a5b728/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/49d13647-137d-4aff-a28b-705df3a5b728/volumes/kubernetes.io~projected/kube-api-access-t857q DeviceMajor:0 DeviceMinor:219 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e3ec7504-ce41-4598-b6e6-1c8706119621/volumes/kubernetes.io~projected/kube-api-access-f6lw5 DeviceMajor:0 DeviceMinor:70 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b813c28-6b1a-429b-a812-5fcffab51b0c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:214 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a5da56c11df534005a197157c1fc916d1247099f25e2097e71fc259848745575/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/eaf49b15-48f0-4abc-8f46-786db6035d5c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:268 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a768d4be81fb4b468cb8900e30700de9f981931893f6cc3338c6fbf3527677f/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/0893ab11-598b-4072-893b-720af09698a7/volumes/kubernetes.io~projected/kube-api-access-jrtcx DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/6ac72c66-33a8-476a-afc7-093dc998aa4e/volumes/kubernetes.io~projected/kube-api-access-46vmp DeviceMajor:0 DeviceMinor:69 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0916d61d034e0ff161bfbf65dd2513fca429fae0930919268a33162db66678e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-199 DeviceMajor:0 DeviceMinor:199 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35844342395beeb62fe5329e7b94effb2052d681fc06b3dd6ee825737025b951/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4ecf31fc9e6e709be10e6a6494f20477229d02b959734e7ebf0b2b3a67aa5f18/userdata/shm DeviceMajor:0 DeviceMinor:105 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/782ad8cb-0365-4cb1-86b0-4d3059401b91/volumes/kubernetes.io~projected/kube-api-access-lnfpb DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd691d7f618be2fe926331b5a40c3120184f382094dd92db39bcda88ad2d2e0d/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b813c28-6b1a-429b-a812-5fcffab51b0c/volumes/kubernetes.io~projected/kube-api-access-nqnvl DeviceMajor:0 DeviceMinor:218 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/1185ece1-18e8-493f-a18c-b367c9793c32/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:224 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0a8442d-746d-4365-a4f5-936b62940823/volumes/kubernetes.io~projected/kube-api-access-2bfp8 DeviceMajor:0 DeviceMinor:269 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f7a33c42-a915-4f9c-bd12-311c839696f5/volumes/kubernetes.io~projected/kube-api-access-n6s75 DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1185ece1-18e8-493f-a18c-b367c9793c32/volumes/kubernetes.io~projected/kube-api-access-7nggt DeviceMajor:0 DeviceMinor:225 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16fdd7f8146325f3689466a415cf4d59c8fdca49806c243300670d5da4b07120/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a595aeb68da09e0e16cce10f78512c1733436de038bb691c8bed81f4d23c949b/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89954462a1cfb3b467a77458afd3eac8bc3b5197f9db6cb8332c87dfe4d569c0/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-103 DeviceMajor:0 DeviceMinor:103 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~projected/kube-api-access-rn5jh DeviceMajor:0 DeviceMinor:68 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/657bec506e881ef6f2c7dc939427d280840c7f02471ddc6c9c6a5e2b32fd83d8/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-99 DeviceMajor:0 DeviceMinor:99 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-121 DeviceMajor:0 DeviceMinor:121 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8936c93b-d868-46be-83e4-b6d224d4b11c/volumes/kubernetes.io~projected/kube-api-access-dkqhg DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d2abeb81-5999-493f-9474-8bf5d01b32df/volumes/kubernetes.io~projected/kube-api-access-8f2qc DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/683fd3fa-45a0-429a-9146-237713d74f5d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/182f9264-f98f-401d-aa1d-001f404fc6af/volumes/kubernetes.io~projected/kube-api-access-2q9qw DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fe38f83-f940-4e6a-8f45-0148745b3693/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:264 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0a768d4be81fb4b MacAddress:b2:62:7f:19:ed:2e Speed:10000 Mtu:8901} {Name:16fdd7f8146325f MacAddress:4e:a2:9a:01:97:19 Speed:10000 Mtu:8901} {Name:7f6ea547fbd9fce MacAddress:46:37:3d:3f:54:a8 Speed:10000 Mtu:8901} {Name:a5da56c11df5340 MacAddress:8a:d9:d6:9e:86:18 Speed:10000 Mtu:8901} {Name:a782cc97fcb81c1 MacAddress:ba:e6:62:c0:d6:77 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c4:83:14:ed Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:77:ff:31:fc:97 Speed:0 Mtu:8901} {Name:d6516779127588e MacAddress:86:ab:12:65:d4:ea Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c4:83:14:ed Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ce:78:e4:ca:58:b4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:a2:44:fd:cd:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954423    4839 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954468    4839 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954748    4839 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954767    4839 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954900    4839 topology_manager.go:143] "Creating topology manager with none policy"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954908    4839 container_manager_linux.go:311] "Creating device plugin manager"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954924    4839 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.954937    4839 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955020    4839 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955222    4839 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955339    4839 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955371    4839 kubelet.go:493] "Attempting to sync node with API server"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955384    4839 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 29 22:42:22.955391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955394    4839 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 29 22:42:22.956084 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955403    4839 kubelet.go:405] "Adding apiserver pod source"
Oct 29 22:42:22.956084 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.955412    4839 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 29 22:42:22.956750 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.956729    4839 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:42:22.956832 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.956763    4839 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:22.957830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.957805    4839 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 29 22:42:22.960286 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960270    4839 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 29 22:42:22.960286 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960284    4839 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 29 22:42:22.960388 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960379    4839 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 29 22:42:22.960577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960566    4839 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 29 22:42:22.960608 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960596    4839 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 29 22:42:22.960707 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960697    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 29 22:42:22.960734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960713    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 29 22:42:22.960734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960719    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 29 22:42:22.960734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960724    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 29 22:42:22.960734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960732    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960742    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960752    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960759    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960766    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960772    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960780    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960790    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960817    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 29 22:42:22.960841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.960826    4839 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 29 22:42:22.964274 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.964260    4839 server.go:1263] "Started kubelet"
Oct 29 22:42:22.964923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.964322    4839 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 29 22:42:22.964923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.964357    4839 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 29 22:42:22.964923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.964521    4839 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 29 22:42:22.964834 ip-10-0-136-187 systemd[1]: Started Kubernetes Kubelet.
Oct 29 22:42:22.966013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.965878    4839 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 29 22:42:22.966013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.965924    4839 server.go:317] "Adding debug handlers to kubelet server"
Oct 29 22:42:22.968413 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.968392    4839 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 29 22:42:22.970602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.968493    4839 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 29 22:42:22.970602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.968575    4839 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-28 22:36:45 +0000 UTC" deadline="2028-07-03 00:59:59.751816033 +0000 UTC"
Oct 29 22:42:22.970602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.968614    4839 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14690h17m36.78320692s"
Oct 29 22:42:22.970602 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:22.968692    4839 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:42:22.970602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.968705    4839 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 29 22:42:22.970602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.968698    4839 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 29 22:42:22.970602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.968756    4839 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 29 22:42:22.972133 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.971245    4839 factory.go:55] Registering systemd factory
Oct 29 22:42:22.972133 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.971267    4839 factory.go:223] Registration of the systemd container factory successfully
Oct 29 22:42:22.973867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973827    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b813c28-6b1a-429b-a812-5fcffab51b0c" volumeName="kubernetes.io/secret/3b813c28-6b1a-429b-a812-5fcffab51b0c-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 29 22:42:22.973952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973884    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="683fd3fa-45a0-429a-9146-237713d74f5d" volumeName="kubernetes.io/projected/683fd3fa-45a0-429a-9146-237713d74f5d-kube-api-access-rn5jh" seLinuxMountContext=""
Oct 29 22:42:22.973952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973900    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="782ad8cb-0365-4cb1-86b0-4d3059401b91" volumeName="kubernetes.io/empty-dir/782ad8cb-0365-4cb1-86b0-4d3059401b91-tmp-dir" seLinuxMountContext=""
Oct 29 22:42:22.973952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973918    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="782ad8cb-0365-4cb1-86b0-4d3059401b91" volumeName="kubernetes.io/projected/782ad8cb-0365-4cb1-86b0-4d3059401b91-kube-api-access-lnfpb" seLinuxMountContext=""
Oct 29 22:42:22.973952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973931    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0893ab11-598b-4072-893b-720af09698a7" volumeName="kubernetes.io/projected/0893ab11-598b-4072-893b-720af09698a7-kube-api-access-jrtcx" seLinuxMountContext=""
Oct 29 22:42:22.973952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973944    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182f9264-f98f-401d-aa1d-001f404fc6af" volumeName="kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-env-overrides" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973961    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182f9264-f98f-401d-aa1d-001f404fc6af" volumeName="kubernetes.io/secret/182f9264-f98f-401d-aa1d-001f404fc6af-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973973    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49d13647-137d-4aff-a28b-705df3a5b728" volumeName="kubernetes.io/empty-dir/49d13647-137d-4aff-a28b-705df3a5b728-tmp-dir" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.973994    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="683fd3fa-45a0-429a-9146-237713d74f5d" volumeName="kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-tmp" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974007    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac72c66-33a8-476a-afc7-093dc998aa4e" volumeName="kubernetes.io/projected/6ac72c66-33a8-476a-afc7-093dc998aa4e-kube-api-access-46vmp" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974023    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac72c66-33a8-476a-afc7-093dc998aa4e" volumeName="kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974035    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8" volumeName="kubernetes.io/configmap/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-konnectivity-ca" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974047    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1185ece1-18e8-493f-a18c-b367c9793c32" volumeName="kubernetes.io/projected/1185ece1-18e8-493f-a18c-b367c9793c32-kube-api-access-7nggt" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974074    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac72c66-33a8-476a-afc7-093dc998aa4e" volumeName="kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974175    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac72c66-33a8-476a-afc7-093dc998aa4e" volumeName="kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-tls" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974194    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8936c93b-d868-46be-83e4-b6d224d4b11c" volumeName="kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:42:22.974204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974205    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8936c93b-d868-46be-83e4-b6d224d4b11c" volumeName="kubernetes.io/projected/8936c93b-d868-46be-83e4-b6d224d4b11c-kube-api-access-dkqhg" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974221    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2abeb81-5999-493f-9474-8bf5d01b32df" volumeName="kubernetes.io/projected/d2abeb81-5999-493f-9474-8bf5d01b32df-kube-api-access-8f2qc" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974233    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ec7504-ce41-4598-b6e6-1c8706119621" volumeName="kubernetes.io/configmap/e3ec7504-ce41-4598-b6e6-1c8706119621-serviceca" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974256    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3ec7504-ce41-4598-b6e6-1c8706119621" volumeName="kubernetes.io/projected/e3ec7504-ce41-4598-b6e6-1c8706119621-kube-api-access-f6lw5" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974273    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49d13647-137d-4aff-a28b-705df3a5b728" volumeName="kubernetes.io/configmap/49d13647-137d-4aff-a28b-705df3a5b728-config-volume" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974286    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fe38f83-f940-4e6a-8f45-0148745b3693" volumeName="kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974302    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182f9264-f98f-401d-aa1d-001f404fc6af" volumeName="kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-config" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974314    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b813c28-6b1a-429b-a812-5fcffab51b0c" volumeName="kubernetes.io/empty-dir/3b813c28-6b1a-429b-a812-5fcffab51b0c-data-volume" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974376    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="683fd3fa-45a0-429a-9146-237713d74f5d" volumeName="kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-etc-tuned" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974396    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8936c93b-d868-46be-83e4-b6d224d4b11c" volumeName="kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-daemon-config" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974420    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaf49b15-48f0-4abc-8f46-786db6035d5c" volumeName="kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974434    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a33c42-a915-4f9c-bd12-311c839696f5" volumeName="kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-binary-copy" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974463    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182f9264-f98f-401d-aa1d-001f404fc6af" volumeName="kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-script-lib" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974478    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182f9264-f98f-401d-aa1d-001f404fc6af" volumeName="kubernetes.io/projected/182f9264-f98f-401d-aa1d-001f404fc6af-kube-api-access-2q9qw" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974496    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49d13647-137d-4aff-a28b-705df3a5b728" volumeName="kubernetes.io/projected/49d13647-137d-4aff-a28b-705df3a5b728-kube-api-access-t857q" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974509    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49d13647-137d-4aff-a28b-705df3a5b728" volumeName="kubernetes.io/secret/49d13647-137d-4aff-a28b-705df3a5b728-metrics-tls" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974526    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac72c66-33a8-476a-afc7-093dc998aa4e" volumeName="kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-metrics-client-ca" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974541    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ac72c66-33a8-476a-afc7-093dc998aa4e" volumeName="kubernetes.io/empty-dir/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-textfile" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974573    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8" volumeName="kubernetes.io/secret/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-agent-certs" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974591    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaf49b15-48f0-4abc-8f46-786db6035d5c" volumeName="kubernetes.io/projected/eaf49b15-48f0-4abc-8f46-786db6035d5c-kube-api-access-bjph9" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974604    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b813c28-6b1a-429b-a812-5fcffab51b0c" volumeName="kubernetes.io/configmap/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 29 22:42:22.974682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974623    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2abeb81-5999-493f-9474-8bf5d01b32df" volumeName="kubernetes.io/configmap/d2abeb81-5999-493f-9474-8bf5d01b32df-iptables-alerter-script" seLinuxMountContext=""
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974786    4839 factory.go:153] Registering CRI-O factory
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974803    4839 factory.go:223] Registration of the crio container factory successfully
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974860    4839 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974895    4839 factory.go:103] Registering Raw factory
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.974910    4839 manager.go:1199] Started watching for new ooms in manager
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975087    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a33c42-a915-4f9c-bd12-311c839696f5" volumeName="kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975106    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a33c42-a915-4f9c-bd12-311c839696f5" volumeName="kubernetes.io/projected/f7a33c42-a915-4f9c-bd12-311c839696f5-kube-api-access-n6s75" seLinuxMountContext=""
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975125    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b813c28-6b1a-429b-a812-5fcffab51b0c" volumeName="kubernetes.io/projected/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-api-access-nqnvl" seLinuxMountContext=""
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975138    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7a33c42-a915-4f9c-bd12-311c839696f5" volumeName="kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975211    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1185ece1-18e8-493f-a18c-b367c9793c32" volumeName="kubernetes.io/secret/1185ece1-18e8-493f-a18c-b367c9793c32-cert" seLinuxMountContext=""
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975230    4839 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0a8442d-746d-4365-a4f5-936b62940823" volumeName="kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8" seLinuxMountContext=""
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975248    4839 reconstruct.go:97] "Volume reconstruction finished"
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975259    4839 reconciler.go:29] "Reconciler: start to sync state"
Oct 29 22:42:22.975641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.975319    4839 manager.go:319] Starting recovery of all containers
Oct 29 22:42:22.999765 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:22.999730    4839 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 29 22:42:23.001094 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.001064    4839 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 29 22:42:23.001094 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.001091    4839 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 29 22:42:23.001314 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.001113    4839 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 29 22:42:23.001314 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.001183    4839 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 29 22:42:23.010457 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.010417    4839 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="09c07ce8a3736f2f0a85d7010f9b16eed909dd6c65fcbcf12b71a9172eba5647" exitCode=0
Oct 29 22:42:23.010457 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.010453    4839 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="cae576831a381f3ee880c045c2c49e8e65cee30ccc025d1404652aa7d68ff8ab" exitCode=0
Oct 29 22:42:23.010649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.010464    4839 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="c5d87a3325225a66f57bd7b955426be267caaad133f2d9b85c5ab38da381bd8b" exitCode=0
Oct 29 22:42:23.010649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.010476    4839 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="1cd66bf33d7ff1952762ea0ebb34e24bc91e056673a45c00009cec998101625a" exitCode=0
Oct 29 22:42:23.010649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.010484    4839 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="400e0b19707f14f3a1d1efdbbb58e41c3674db739f7bbc6555ff28f6d24bf9d2" exitCode=0
Oct 29 22:42:23.010649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.010493    4839 generic.go:356] "Generic (PLEG): container finished" podID="f7a33c42-a915-4f9c-bd12-311c839696f5" containerID="421d536c8667ee94c3fecc189e914f6ad09fc1a359c8540b1b60f64a9833a715" exitCode=0
Oct 29 22:42:23.012423 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.012396    4839 generic.go:356] "Generic (PLEG): container finished" podID="a7f8c8d4e5e3caee624106946868ca7a" containerID="49b9963d8b2c8c1be521159d6d6da1eebdab546ad08f7f7ed13d180c9f6d98b4" exitCode=0
Oct 29 22:42:23.023527 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.023503    4839 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 29 22:42:23.039289 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.039166    4839 generic.go:356] "Generic (PLEG): container finished" podID="6ac72c66-33a8-476a-afc7-093dc998aa4e" containerID="4c36de1b23ae607076d48ad781ef960e7fec8eb023876e1789b84fa2db9c5bd8" exitCode=0
Oct 29 22:42:23.054309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.054286    4839 manager.go:324] Recovery completed
Oct 29 22:42:23.068775 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.068746    4839 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:42:23.069603 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.069590    4839 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:23.070785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.070765    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:23.070954 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.070794    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:23.070954 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.070804    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:23.072965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.072948    4839 cpu_manager.go:226] "Starting" policy="none"
Oct 29 22:42:23.073041 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.072964    4839 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 29 22:42:23.073041 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.072966    4839 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:23.073041 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.072987    4839 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 29 22:42:23.073220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.073141    4839 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 29 22:42:23.073265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.073221    4839 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 29 22:42:23.073265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.073239    4839 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 29 22:42:23.073265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.073245    4839 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 29 22:42:23.073265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.073252    4839 policy_none.go:50] "Start"
Oct 29 22:42:23.073265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.073261    4839 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 29 22:42:23.073397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.073272    4839 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 29 22:42:23.074690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.074676    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:23.074752 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.074702    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:23.074752 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.074712    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:23.076731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.076694    4839 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 29 22:42:23.076731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.076711    4839 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 29 22:42:23.076731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.076718    4839 policy_none.go:44] "Start"
Oct 29 22:42:23.082193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.082163    4839 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 29 22:42:23.085934 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.085922    4839 manager.go:341] "Starting Device Plugin manager"
Oct 29 22:42:23.085979 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.085955    4839 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 29 22:42:23.085979 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.085965    4839 server.go:87] "Starting device plugin registration server"
Oct 29 22:42:23.086291 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.086279    4839 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 29 22:42:23.086324 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.086293    4839 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 29 22:42:23.086400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.086375    4839 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 29 22:42:23.086532 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.086493    4839 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 29 22:42:23.086596 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.086538    4839 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 29 22:42:23.086596 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.086545    4839 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 29 22:42:23.087399 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.086618    4839 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-29T22:42:23.08652696Z","UUID":"391a3db3-01b1-48d4-a66e-7b5e8e73f51d","Handler":null,"Name":"","Endpoint":""}
Oct 29 22:42:23.087863 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.087846    4839 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 29 22:42:23.087932 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.087878    4839 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-187.ec2.internal\" not found"
Oct 29 22:42:23.088764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.088722    4839 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 29 22:42:23.088764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.088759    4839 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 29 22:42:23.088943 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.088833    4839 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.089389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.089344    4839 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.102302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.102267    4839 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"]
Oct 29 22:42:23.102868 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.102854    4839 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.102903 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.102887    4839 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.102936 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.102886    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal" event={"ID":"a7f8c8d4e5e3caee624106946868ca7a","Type":"ContainerStarted","Data":"fe6889054d29cadbc2f606e9cb0c0db5f3a1bc4a0cf31e0e4d2207c3be877e05"}
Oct 29 22:42:23.102970 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.102951    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal" event={"ID":"a7f8c8d4e5e3caee624106946868ca7a","Type":"ContainerDied","Data":"49b9963d8b2c8c1be521159d6d6da1eebdab546ad08f7f7ed13d180c9f6d98b4"}
Oct 29 22:42:23.103001 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.102975    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal" event={"ID":"a7f8c8d4e5e3caee624106946868ca7a","Type":"ContainerStarted","Data":"2c3c15943d663cbee60c23f10aa65ed94b57fb4822814e366c5dc1e7182d6516"}
Oct 29 22:42:23.103077 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.103064    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal" event={"ID":"5a74b888ca6cfc8b230ed1966a77b369","Type":"ContainerStarted","Data":"fd428a0255091019f8efbb20b192446d6ec3ea61a9461b3e8fec68552b6ed156"}
Oct 29 22:42:23.103134 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.103083    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal" event={"ID":"5a74b888ca6cfc8b230ed1966a77b369","Type":"ContainerStarted","Data":"d194e280ebf0874e449ab87e823fcfcf1846ef1ae1c563a0cb9e6050719825b3"}
Oct 29 22:42:23.103395 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.103380    4839 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.103695 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.103681    4839 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 29 22:42:23.125261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.125241    4839 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:23.125350 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.125281    4839 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.128445 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.128432    4839 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:23.128525 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.128457    4839 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.169304 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.169274    4839 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.187234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.187215    4839 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 29 22:42:23.188429 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.188411    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientMemory"
Oct 29 22:42:23.188487 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.188441    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasNoDiskPressure"
Oct 29 22:42:23.188487 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.188453    4839 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-187.ec2.internal" event="NodeHasSufficientPID"
Oct 29 22:42:23.188540 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.188494    4839 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.193189 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.193173    4839 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:23.193239 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.193211    4839 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.193239 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.193222    4839 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.217672 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.217653    4839 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:23.217712 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:23.217685    4839 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.224848 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.224827    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.224897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.224854    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.224897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.224873    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a74b888ca6cfc8b230ed1966a77b369-config\") pod \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" (UID: \"5a74b888ca6cfc8b230ed1966a77b369\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.228333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.228320    4839 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.228388 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.228380    4839 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.326023 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.325987    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.326023 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.326020    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.326238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.326036    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a74b888ca6cfc8b230ed1966a77b369-config\") pod \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" (UID: \"5a74b888ca6cfc8b230ed1966a77b369\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.326238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.326097    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.326238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.326163    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f8c8d4e5e3caee624106946868ca7a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" (UID: \"a7f8c8d4e5e3caee624106946868ca7a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.326238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.326208    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5a74b888ca6cfc8b230ed1966a77b369-config\") pod \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" (UID: \"5a74b888ca6cfc8b230ed1966a77b369\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:23.956644 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:23.956607    4839 apiserver.go:52] "Watching apiserver"
Oct 29 22:42:24.002536 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.002514    4839 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 29 22:42:24.002952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.002933    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal","openshift-network-operator/iptables-alerter-ltvq9","openshift-monitoring/node-exporter-6jfhv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r","openshift-ingress-canary/ingress-canary-k7l64","openshift-network-diagnostics/network-check-target-m7w8j","openshift-cluster-node-tuning-operator/tuned-6ltpj","openshift-dns/dns-default-692j2","openshift-image-registry/node-ca-qxjxd","openshift-multus/multus-additional-cni-plugins-w2wcc","openshift-dns/node-resolver-rhshl","openshift-multus/network-metrics-daemon-6nqg4","kube-system/global-pull-secret-syncer-cn2r6","kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal","openshift-insights/insights-runtime-extractor-kf7bp","openshift-multus/multus-zksnr","openshift-ovn-kubernetes/ovnkube-node-z5wst","kube-system/konnectivity-agent-dwfrq"]
Oct 29 22:42:24.019860 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.019829    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4f7f9\""
Oct 29 22:42:24.019999 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.019901    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 29 22:42:24.020364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.020347    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.020407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.020374    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.020931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.020917    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bd868\""
Oct 29 22:42:24.024991 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.024953    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 29 22:42:24.025119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025003    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 29 22:42:24.025119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025009    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.025119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025020    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.025296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025119    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.025296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025122    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 29 22:42:24.025296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025165    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.025296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025203    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-56msd\""
Oct 29 22:42:24.025296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025207    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 29 22:42:24.025296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025231    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 29 22:42:24.025459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025358    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kfh6g\""
Oct 29 22:42:24.025459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025395    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 29 22:42:24.025524 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.025474    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7wdgt\""
Oct 29 22:42:24.033695 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.033672    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:24.036637 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.036623    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.036741 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.036704    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 29 22:42:24.036803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.036745    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.036803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.036778    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.037311 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037298    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 29 22:42:24.037311 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037307    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 29 22:42:24.037509 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037497    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 29 22:42:24.037588 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037509    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-6bbfq\""
Oct 29 22:42:24.037630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037603    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.037630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037601    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 29 22:42:24.037721 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037706    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 29 22:42:24.037842 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037831    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.037885 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037863    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jmngh\""
Oct 29 22:42:24.037927 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037912    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gqfpm\""
Oct 29 22:42:24.037991 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.037952    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7jzk\""
Oct 29 22:42:24.038045 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.038015    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g5fdr\""
Oct 29 22:42:24.038160 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.038130    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 29 22:42:24.038200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.038130    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qj89s\""
Oct 29 22:42:24.038200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.038181    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pdbx4\""
Oct 29 22:42:24.038200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.038184    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p7zhx\""
Oct 29 22:42:24.038997 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.038983    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 29 22:42:24.039059 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.039046    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-528cc\""
Oct 29 22:42:24.039092 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.039066    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8pxqn\""
Oct 29 22:42:24.039262 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.039246    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 29 22:42:24.039336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.039322    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9vstz\""
Oct 29 22:42:24.039930 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.039915    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.039985 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.039974    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.040026 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.039983    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.040237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.040224    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.040555 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.040544    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.041568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041552    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 29 22:42:24.041568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041564    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 29 22:42:24.041635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041554    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.041635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041602    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 29 22:42:24.041635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041621    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 29 22:42:24.041718 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041629    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.041893 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041878    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 29 22:42:24.041942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041894    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 29 22:42:24.041942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041905    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.041942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041905    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.041942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041927    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 29 22:42:24.041942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041940    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 29 22:42:24.042092 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.041908    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 29 22:42:24.042613 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.042600    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 29 22:42:24.047537 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.047518    4839 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:24.047613 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.047598    4839 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:24.070376 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.070358    4839 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 29 22:42:24.075781 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.075768    4839 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:24.075824 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:24.075814    4839 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-187.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:24.076616 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.076604    4839 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 29 22:42:24.076651 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:24.076638    4839 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-187.ec2.internal"
Oct 29 22:42:24.129264 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129234    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.129467 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129273    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-netns\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.129467 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129306    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-hostroot\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.129467 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129330    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2abeb81-5999-493f-9474-8bf5d01b32df-iptables-alerter-script\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:42:24.129467 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129389    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-systemd-units\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.129649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129496    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.129649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129535    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-log-socket\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.129649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129590    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnfpb\" (UniqueName: \"kubernetes.io/projected/782ad8cb-0365-4cb1-86b0-4d3059401b91-kube-api-access-lnfpb\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:42:24.129649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129624    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-tls\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.129815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129651    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-kubelet\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.129815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129674    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-device-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.129815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129689    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-accelerators-collector-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.129815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129696    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d2abeb81-5999-493f-9474-8bf5d01b32df-iptables-alerter-script\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:42:24.129815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129700    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.129815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129752    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqnvl\" (UniqueName: \"kubernetes.io/projected/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-api-access-nqnvl\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.130072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129819    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-conf-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.130072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129900    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-tls\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.130072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129994    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d13647-137d-4aff-a28b-705df3a5b728-metrics-tls\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.130072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.129991    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.130072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130024    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-socket-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.130072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130050    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2abeb81-5999-493f-9474-8bf5d01b32df-host-slash\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130086    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2qc\" (UniqueName: \"kubernetes.io/projected/d2abeb81-5999-493f-9474-8bf5d01b32df-kube-api-access-8f2qc\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130125    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130182    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6s75\" (UniqueName: \"kubernetes.io/projected/f7a33c42-a915-4f9c-bd12-311c839696f5-kube-api-access-n6s75\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130213    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-node-log\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130189    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/49d13647-137d-4aff-a28b-705df3a5b728-metrics-tls\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130238    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-script-lib\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130278    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782ad8cb-0365-4cb1-86b0-4d3059401b91-hosts-file\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130302    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782ad8cb-0365-4cb1-86b0-4d3059401b91-tmp-dir\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:42:24.130333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130325    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-host\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130322    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130352    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130415    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-agent-certs\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130461    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-netd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130478    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/782ad8cb-0365-4cb1-86b0-4d3059401b91-tmp-dir\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130478    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-script-lib\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130487    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-env-overrides\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130547    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.130612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130594    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-var-lib-kubelet\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130619    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-bin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130622    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-env-overrides\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130642    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130675    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-sys-fs\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130722    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3ec7504-ce41-4598-b6e6-1c8706119621-serviceca\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130736    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fe38f83-f940-4e6a-8f45-0148745b3693-original-pull-secret\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130744    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-agent-certs\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130776    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6lw5\" (UniqueName: \"kubernetes.io/projected/e3ec7504-ce41-4598-b6e6-1c8706119621-kube-api-access-f6lw5\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:42:24.130931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.130838    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-sys\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.131294 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131224    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-textfile\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.131294 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131271    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.131389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131316    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e3ec7504-ce41-4598-b6e6-1c8706119621-serviceca\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:42:24.131389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131337    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-etc-tuned\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.131389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131372    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d13647-137d-4aff-a28b-705df3a5b728-tmp-dir\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.131513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131452    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49d13647-137d-4aff-a28b-705df3a5b728-tmp-dir\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.131513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131452    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-etc-tuned\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.131513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131467    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-textfile\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.131513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131506    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtcx\" (UniqueName: \"kubernetes.io/projected/0893ab11-598b-4072-893b-720af09698a7-kube-api-access-jrtcx\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.131684 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131638    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.131684 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131640    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1185ece1-18e8-493f-a18c-b367c9793c32-cert\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:24.131778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131763    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-kubelet-config\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.131823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131790    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-tmp\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.131823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131814    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-system-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.131930 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131888    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1185ece1-18e8-493f-a18c-b367c9793c32-cert\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:24.131930 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131906    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.131930 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131894    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/683fd3fa-45a0-429a-9146-237713d74f5d-tmp\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131933    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-socket-dir-parent\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.132082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.131999    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjph9\" (UniqueName: \"kubernetes.io/projected/eaf49b15-48f0-4abc-8f46-786db6035d5c-kube-api-access-bjph9\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:24.132082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132038    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-systemd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.132082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132064    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-etc-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.132302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132095    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132127    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-conf\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132188    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-run\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132216    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-cni-binary-copy\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.132302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132247    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-registration-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.132302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132284    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-config\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132319    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vmp\" (UniqueName: \"kubernetes.io/projected/6ac72c66-33a8-476a-afc7-093dc998aa4e-kube-api-access-46vmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132349    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-lib-modules\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132384    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-var-lib-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132412    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-system-cni-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132439    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3ec7504-ce41-4598-b6e6-1c8706119621-host\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132470    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-root\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132509    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.132560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132541    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b813c28-6b1a-429b-a812-5fcffab51b0c-data-volume\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132571    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-modprobe-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132604    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5jh\" (UniqueName: \"kubernetes.io/projected/683fd3fa-45a0-429a-9146-237713d74f5d-kube-api-access-rn5jh\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132636    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-ovn\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132667    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132696    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-metrics-client-ca\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132726    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysconfig\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132757    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-sys\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132786    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-cnibin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132810    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-k8s-cni-cncf-io\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132826    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-multus-certs\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132847    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-cnibin\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.132886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132877    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132897    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132918    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b813c28-6b1a-429b-a812-5fcffab51b0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132939    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-dbus\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132958    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-daemon-config\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132975    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.132997    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-wtmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133017    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-systemd\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133037    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqhg\" (UniqueName: \"kubernetes.io/projected/8936c93b-d868-46be-83e4-b6d224d4b11c-kube-api-access-dkqhg\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133054    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t857q\" (UniqueName: \"kubernetes.io/projected/49d13647-137d-4aff-a28b-705df3a5b728-kube-api-access-t857q\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133074    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133094    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-konnectivity-ca\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133113    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-kubelet\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133132    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-slash\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133176    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q9qw\" (UniqueName: \"kubernetes.io/projected/182f9264-f98f-401d-aa1d-001f404fc6af-kube-api-access-2q9qw\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133210    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-etc-kubernetes\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133240    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-bin\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133270    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/182f9264-f98f-401d-aa1d-001f404fc6af-ovn-node-metrics-cert\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133298    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b813c28-6b1a-429b-a812-5fcffab51b0c-crio-socket\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133324    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-kubernetes\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133353    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-multus\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.133401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133385    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d13647-137d-4aff-a28b-705df3a5b728-config-volume\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.134240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133414    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nggt\" (UniqueName: \"kubernetes.io/projected/1185ece1-18e8-493f-a18c-b367c9793c32-kube-api-access-7nggt\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:24.134240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133447    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.134240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.133471    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-netns\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.134240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.134080    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-cni-binary-copy\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.134415 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.134329    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/182f9264-f98f-401d-aa1d-001f404fc6af-ovnkube-config\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.134847 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.134826    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.134933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.134917    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b813c28-6b1a-429b-a812-5fcffab51b0c-data-volume\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.135515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.135490    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ac72c66-33a8-476a-afc7-093dc998aa4e-metrics-client-ca\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.135738 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.135715    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/49d13647-137d-4aff-a28b-705df3a5b728-config-volume\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.135807 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.135744    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3b813c28-6b1a-429b-a812-5fcffab51b0c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.135807 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.135774    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8-konnectivity-ca\") pod \"konnectivity-agent-dwfrq\" (UID: \"9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8\") " pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:42:24.135984 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.135967    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-daemon-config\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.136046 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.135995    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/182f9264-f98f-401d-aa1d-001f404fc6af-ovn-node-metrics-cert\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.136285 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.136265    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eaf49b15-48f0-4abc-8f46-786db6035d5c-metrics-certs\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:24.136364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.136320    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7a33c42-a915-4f9c-bd12-311c839696f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.145301 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.145279    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:42:24.149779 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.149755    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqnvl\" (UniqueName: \"kubernetes.io/projected/3b813c28-6b1a-429b-a812-5fcffab51b0c-kube-api-access-nqnvl\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.149952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.149932    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2qc\" (UniqueName: \"kubernetes.io/projected/d2abeb81-5999-493f-9474-8bf5d01b32df-kube-api-access-8f2qc\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:42:24.150032 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.149997    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6s75\" (UniqueName: \"kubernetes.io/projected/f7a33c42-a915-4f9c-bd12-311c839696f5-kube-api-access-n6s75\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.150398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.150374    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnfpb\" (UniqueName: \"kubernetes.io/projected/782ad8cb-0365-4cb1-86b0-4d3059401b91-kube-api-access-lnfpb\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:42:24.152796 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.152767    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6lw5\" (UniqueName: \"kubernetes.io/projected/e3ec7504-ce41-4598-b6e6-1c8706119621-kube-api-access-f6lw5\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:42:24.156379 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.156361    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vmp\" (UniqueName: \"kubernetes.io/projected/6ac72c66-33a8-476a-afc7-093dc998aa4e-kube-api-access-46vmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.157237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.157217    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjph9\" (UniqueName: \"kubernetes.io/projected/eaf49b15-48f0-4abc-8f46-786db6035d5c-kube-api-access-bjph9\") pod \"network-metrics-daemon-6nqg4\" (UID: \"eaf49b15-48f0-4abc-8f46-786db6035d5c\") " pod="openshift-multus/network-metrics-daemon-6nqg4"
Oct 29 22:42:24.158507 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.158488    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtcx\" (UniqueName: \"kubernetes.io/projected/0893ab11-598b-4072-893b-720af09698a7-kube-api-access-jrtcx\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.159218 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.159203    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t857q\" (UniqueName: \"kubernetes.io/projected/49d13647-137d-4aff-a28b-705df3a5b728-kube-api-access-t857q\") pod \"dns-default-692j2\" (UID: \"49d13647-137d-4aff-a28b-705df3a5b728\") " pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:24.166214 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.166194    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfp8\" (UniqueName: \"kubernetes.io/projected/f0a8442d-746d-4365-a4f5-936b62940823-kube-api-access-2bfp8\") pod \"network-check-target-m7w8j\" (UID: \"f0a8442d-746d-4365-a4f5-936b62940823\") " pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:24.166297 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.166257    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q9qw\" (UniqueName: \"kubernetes.io/projected/182f9264-f98f-401d-aa1d-001f404fc6af-kube-api-access-2q9qw\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.168731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.168495    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqhg\" (UniqueName: \"kubernetes.io/projected/8936c93b-d868-46be-83e4-b6d224d4b11c-kube-api-access-dkqhg\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.168731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.168676    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5jh\" (UniqueName: \"kubernetes.io/projected/683fd3fa-45a0-429a-9146-237713d74f5d-kube-api-access-rn5jh\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.168877 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.168845    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nggt\" (UniqueName: \"kubernetes.io/projected/1185ece1-18e8-493f-a18c-b367c9793c32-kube-api-access-7nggt\") pod \"ingress-canary-k7l64\" (UID: \"1185ece1-18e8-493f-a18c-b367c9793c32\") " pod="openshift-ingress-canary/ingress-canary-k7l64"
Oct 29 22:42:24.234725 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234645    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-conf-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.234725 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234676    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-socket-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.234725 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234692    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2abeb81-5999-493f-9474-8bf5d01b32df-host-slash\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:42:24.234725 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234708    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-node-log\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.234725 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234728    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782ad8cb-0365-4cb1-86b0-4d3059401b91-hosts-file\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234762    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-host\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234763    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-conf-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234781    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2abeb81-5999-493f-9474-8bf5d01b32df-host-slash\") pod \"iptables-alerter-ltvq9\" (UID: \"d2abeb81-5999-493f-9474-8bf5d01b32df\") " pod="openshift-network-operator/iptables-alerter-ltvq9"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234792    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-socket-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234787    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234806    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-node-log\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234835    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234837    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-host\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234832    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/782ad8cb-0365-4cb1-86b0-4d3059401b91-hosts-file\") pod \"node-resolver-rhshl\" (UID: \"782ad8cb-0365-4cb1-86b0-4d3059401b91\") " pod="openshift-dns/node-resolver-rhshl"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234848    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-netd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234873    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-var-lib-kubelet\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234870    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-netd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234895    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-bin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234917    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234933    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-sys-fs\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234936    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-var-lib-kubelet\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.234962    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-sys\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235007    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-bin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235007    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235016    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-sys-fs\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235025    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-kubelet-config\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235011    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-sys\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235054    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-system-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235079    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235100    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-socket-dir-parent\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235106    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-kubelet-config\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235108    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-system-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235124    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-systemd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235166    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-socket-dir-parent\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235168    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-etc-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235174    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-multus-cni-dir\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235186    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-systemd\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235191    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235209    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-etc-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235218    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-conf\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235250    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-run\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235278    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-registration-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235294    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-lib-modules\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235317    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-var-lib-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235327    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235333    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-system-cni-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235349    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3ec7504-ce41-4598-b6e6-1c8706119621-host\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235356    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysctl-conf\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235375    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-run\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235374    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-root\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235386    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-system-cni-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235374    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-registration-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235404    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-modprobe-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235409    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-root\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235400    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e3ec7504-ce41-4598-b6e6-1c8706119621-host\") pod \"node-ca-qxjxd\" (UID: \"e3ec7504-ce41-4598-b6e6-1c8706119621\") " pod="openshift-image-registry/node-ca-qxjxd"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235422    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-lib-modules\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235405    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-var-lib-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235437    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-ovn\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235456    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-ovn\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235466    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-modprobe-d\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235495    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235517    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysconfig\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235534    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-sys\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235550    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-cnibin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235584    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-sysconfig\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235598    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-sys\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235604    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-cnibin\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235604    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235629    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-k8s-cni-cncf-io\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235651    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-multus-certs\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235656    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-k8s-cni-cncf-io\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235667    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-cnibin\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235682    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-dbus\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235698    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235703    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-multus-certs\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235714    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-wtmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235744    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235743    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-systemd\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235780    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235806    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-cnibin\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235807    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-kubelet\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235824    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6ac72c66-33a8-476a-afc7-093dc998aa4e-node-exporter-wtmp\") pod \"node-exporter-6jfhv\" (UID: \"6ac72c66-33a8-476a-afc7-093dc998aa4e\") " pod="openshift-monitoring/node-exporter-6jfhv"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235812    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fe38f83-f940-4e6a-8f45-0148745b3693-dbus\") pod \"global-pull-secret-syncer-cn2r6\" (UID: \"0fe38f83-f940-4e6a-8f45-0148745b3693\") " pod="kube-system/global-pull-secret-syncer-cn2r6"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235838    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-systemd\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235843    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-kubelet\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235850    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-slash\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235855    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7a33c42-a915-4f9c-bd12-311c839696f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w2wcc\" (UID: \"f7a33c42-a915-4f9c-bd12-311c839696f5\") " pod="openshift-multus/multus-additional-cni-plugins-w2wcc"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235877    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-slash\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235889    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-etc-kubernetes\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235910    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-bin\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235925    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b813c28-6b1a-429b-a812-5fcffab51b0c-crio-socket\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235940    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-kubernetes\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235965    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-multus\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235976    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/683fd3fa-45a0-429a-9146-237713d74f5d-etc-kubernetes\") pod \"tuned-6ltpj\" (UID: \"683fd3fa-45a0-429a-9146-237713d74f5d\") " pod="openshift-cluster-node-tuning-operator/tuned-6ltpj"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235996    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-netns\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.235998    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3b813c28-6b1a-429b-a812-5fcffab51b0c-crio-socket\") pod \"insights-runtime-extractor-kf7bp\" (UID: \"3b813c28-6b1a-429b-a812-5fcffab51b0c\") " pod="openshift-insights/insights-runtime-extractor-kf7bp"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236000    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-cni-bin\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.235994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236018    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-etc-kubernetes\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236045    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-host-run-netns\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236046    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-netns\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236065    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-run-netns\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236072    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-hostroot\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236092    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-hostroot\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236098    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-systemd-units\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236118    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-cni-multus\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236142    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236180    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-log-socket\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236204    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-systemd-units\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236224    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-run-openvswitch\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236251    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-kubelet\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236264    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/182f9264-f98f-401d-aa1d-001f404fc6af-log-socket\") pod \"ovnkube-node-z5wst\" (UID: \"182f9264-f98f-401d-aa1d-001f404fc6af\") " pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236291    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-device-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236313    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8936c93b-d868-46be-83e4-b6d224d4b11c-host-var-lib-kubelet\") pod \"multus-zksnr\" (UID: \"8936c93b-d868-46be-83e4-b6d224d4b11c\") " pod="openshift-multus/multus-zksnr"
Oct 29 22:42:24.237816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.236333    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0893ab11-598b-4072-893b-720af09698a7-device-dir\") pod \"aws-ebs-csi-driver-node-rtc9r\" (UID: \"0893ab11-598b-4072-893b-720af09698a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rtc9r"
Oct 29 22:42:24.870945 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:24.870915    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dwfrq"
Oct 29 22:42:27.089919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:27.089891    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:27.091513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:27.091487    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-692j2"
Oct 29 22:42:28.922317 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:28.922272    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:42:29.659260 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:29.659235    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:33.236722 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:42:33.236677    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:42:34.298818 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:34.298791    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:34.299180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:34.299012    4839 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 29 22:42:38.559898 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:38.559868    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z5wst"
Oct 29 22:42:58.925491 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:42:58.925458    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m7w8j"
Oct 29 22:43:03.236793 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:43:03.236759    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:43:13.330094 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:13.330015    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:33.235616 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:43:33.235573    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:43:55.733745 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.733711    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"]
Oct 29 22:43:55.736696 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.736678    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.755666 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.755638    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"]
Oct 29 22:43:55.769827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.769791    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:43:55.769827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.769817    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-6gsp7\""
Oct 29 22:43:55.769827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.769800    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:43:55.784316 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.784285    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:55.794228 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.794203    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.794326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.794236    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b003203c-0157-432a-9755-cffa4e6e3d30-kube-api-access-kghd6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.794373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.794356    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.865464 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.865416    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:43:55.895659 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.895628    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.895771 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.895676    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.895771 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.895694    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b003203c-0157-432a-9755-cffa4e6e3d30-kube-api-access-kghd6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.896106 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.896086    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.896220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.896126    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:55.914609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.914584    4839 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 29 22:43:55.916794 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:55.916769    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b003203c-0157-432a-9755-cffa4e6e3d30-kube-api-access-kghd6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:56.051717 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:56.051687    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:43:56.341005 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:56.340977    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"]
Oct 29 22:43:56.342305 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:43:56.342252    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb003203c_0157_432a_9755_cffa4e6e3d30.slice/crio-54c77711b0fbafc5a00c4b5a8db9786c962e1e786b72a88d86ebda05b90da2d4 WatchSource:0}: Error finding container 54c77711b0fbafc5a00c4b5a8db9786c962e1e786b72a88d86ebda05b90da2d4: Status 404 returned error can't find the container with id 54c77711b0fbafc5a00c4b5a8db9786c962e1e786b72a88d86ebda05b90da2d4
Oct 29 22:43:56.344365 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:56.344346    4839 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:43:57.322010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:43:57.321963    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58" event={"ID":"b003203c-0157-432a-9755-cffa4e6e3d30","Type":"ContainerStarted","Data":"54c77711b0fbafc5a00c4b5a8db9786c962e1e786b72a88d86ebda05b90da2d4"}
Oct 29 22:44:02.339462 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:02.339425    4839 generic.go:356] "Generic (PLEG): container finished" podID="b003203c-0157-432a-9755-cffa4e6e3d30" containerID="1dfb1eb327e9a8fce28287e0086ff705b81a3e1617ce421c088b2512aa3f6798" exitCode=0
Oct 29 22:44:02.339876 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:02.339481    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58" event={"ID":"b003203c-0157-432a-9755-cffa4e6e3d30","Type":"ContainerDied","Data":"1dfb1eb327e9a8fce28287e0086ff705b81a3e1617ce421c088b2512aa3f6798"}
Oct 29 22:44:03.240604 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:44:03.240564    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:44:04.347709 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:04.347677    4839 generic.go:356] "Generic (PLEG): container finished" podID="b003203c-0157-432a-9755-cffa4e6e3d30" containerID="356201ade9fea6c436da1d3b6612894badae4567989540ec8bd2fbb39155fefa" exitCode=0
Oct 29 22:44:04.348089 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:04.347723    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58" event={"ID":"b003203c-0157-432a-9755-cffa4e6e3d30","Type":"ContainerDied","Data":"356201ade9fea6c436da1d3b6612894badae4567989540ec8bd2fbb39155fefa"}
Oct 29 22:44:11.372109 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:11.372069    4839 generic.go:356] "Generic (PLEG): container finished" podID="b003203c-0157-432a-9755-cffa4e6e3d30" containerID="09ae50607c5fdc4df44e406b71f47787497a406e1506304e8b15660620f74bad" exitCode=0
Oct 29 22:44:11.372523 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:11.372166    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58" event={"ID":"b003203c-0157-432a-9755-cffa4e6e3d30","Type":"ContainerDied","Data":"09ae50607c5fdc4df44e406b71f47787497a406e1506304e8b15660620f74bad"}
Oct 29 22:44:12.152899 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.152867    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:12.491398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.491371    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:44:12.607266 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.607212    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b003203c-0157-432a-9755-cffa4e6e3d30-kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b003203c-0157-432a-9755-cffa4e6e3d30-kube-api-access-kghd6\") pod \"b003203c-0157-432a-9755-cffa4e6e3d30\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") "
Oct 29 22:44:12.607266 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.607275    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-util\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-util\") pod \"b003203c-0157-432a-9755-cffa4e6e3d30\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") "
Oct 29 22:44:12.607534 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.607297    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-bundle\") pod \"b003203c-0157-432a-9755-cffa4e6e3d30\" (UID: \"b003203c-0157-432a-9755-cffa4e6e3d30\") "
Oct 29 22:44:12.607925 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.607891    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-bundle" pod "b003203c-0157-432a-9755-cffa4e6e3d30" (UID: "b003203c-0157-432a-9755-cffa4e6e3d30"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:12.609567 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.609541    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b003203c-0157-432a-9755-cffa4e6e3d30-kube-api-access-kghd6" pod "b003203c-0157-432a-9755-cffa4e6e3d30" (UID: "b003203c-0157-432a-9755-cffa4e6e3d30"). InnerVolumeSpecName "kube-api-access-kghd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:12.611737 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.611712    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-util" pod "b003203c-0157-432a-9755-cffa4e6e3d30" (UID: "b003203c-0157-432a-9755-cffa4e6e3d30"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:12.708506 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.708410    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kghd6\" (UniqueName: \"kubernetes.io/projected/b003203c-0157-432a-9755-cffa4e6e3d30-kube-api-access-kghd6\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:12.708506 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.708452    4839 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-util\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:12.708506 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:12.708462    4839 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b003203c-0157-432a-9755-cffa4e6e3d30-bundle\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:13.383388 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:13.383350    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58" event={"ID":"b003203c-0157-432a-9755-cffa4e6e3d30","Type":"ContainerDied","Data":"54c77711b0fbafc5a00c4b5a8db9786c962e1e786b72a88d86ebda05b90da2d4"}
Oct 29 22:44:13.383556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:13.383389    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c44m58"
Oct 29 22:44:13.383556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:13.383424    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54c77711b0fbafc5a00c4b5a8db9786c962e1e786b72a88d86ebda05b90da2d4"
Oct 29 22:44:13.443197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:13.443167    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:16.123230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:16.123200    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:16.224451 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:16.224424    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:16.773692 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:16.773661    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:17.611691 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:17.611659    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:18.802620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802589    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"]
Oct 29 22:44:18.802998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802795    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b003203c-0157-432a-9755-cffa4e6e3d30" containerName="util"
Oct 29 22:44:18.802998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802805    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b003203c-0157-432a-9755-cffa4e6e3d30" containerName="util"
Oct 29 22:44:18.802998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802812    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b003203c-0157-432a-9755-cffa4e6e3d30" containerName="extract"
Oct 29 22:44:18.802998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802817    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b003203c-0157-432a-9755-cffa4e6e3d30" containerName="extract"
Oct 29 22:44:18.802998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802828    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b003203c-0157-432a-9755-cffa4e6e3d30" containerName="pull"
Oct 29 22:44:18.802998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802833    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b003203c-0157-432a-9755-cffa4e6e3d30" containerName="pull"
Oct 29 22:44:18.802998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.802873    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b003203c-0157-432a-9755-cffa4e6e3d30" containerName="extract"
Oct 29 22:44:18.805875 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.805857    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:18.815635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.815610    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"]
Oct 29 22:44:18.829583 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.829558    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 29 22:44:18.829704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.829611    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:44:18.829704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.829611    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6h7bn\""
Oct 29 22:44:18.847921 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.847899    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0e70adf-7115-4e24-a2f7-e059aee572b2-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-2scmh\" (UID: \"f0e70adf-7115-4e24-a2f7-e059aee572b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:18.848004 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.847928    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzlt\" (UniqueName: \"kubernetes.io/projected/f0e70adf-7115-4e24-a2f7-e059aee572b2-kube-api-access-xzzlt\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-2scmh\" (UID: \"f0e70adf-7115-4e24-a2f7-e059aee572b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:18.948393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.948356    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0e70adf-7115-4e24-a2f7-e059aee572b2-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-2scmh\" (UID: \"f0e70adf-7115-4e24-a2f7-e059aee572b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:18.948393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.948397    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzlt\" (UniqueName: \"kubernetes.io/projected/f0e70adf-7115-4e24-a2f7-e059aee572b2-kube-api-access-xzzlt\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-2scmh\" (UID: \"f0e70adf-7115-4e24-a2f7-e059aee572b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:18.948733 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.948712    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0e70adf-7115-4e24-a2f7-e059aee572b2-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-2scmh\" (UID: \"f0e70adf-7115-4e24-a2f7-e059aee572b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:18.973590 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:18.973563    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzlt\" (UniqueName: \"kubernetes.io/projected/f0e70adf-7115-4e24-a2f7-e059aee572b2-kube-api-access-xzzlt\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-2scmh\" (UID: \"f0e70adf-7115-4e24-a2f7-e059aee572b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:19.116891 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:19.116809    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"
Oct 29 22:44:19.433194 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:19.433105    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh"]
Oct 29 22:44:19.445663 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:44:19.445635    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e70adf_7115_4e24_a2f7_e059aee572b2.slice/crio-1fad9ba053039f304205bf422c7869edc8799ab3280d01c84c9dc81debe32302 WatchSource:0}: Error finding container 1fad9ba053039f304205bf422c7869edc8799ab3280d01c84c9dc81debe32302: Status 404 returned error can't find the container with id 1fad9ba053039f304205bf422c7869edc8799ab3280d01c84c9dc81debe32302
Oct 29 22:44:19.989681 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:19.989651    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:20.094332 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:20.094295    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:20.409191 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:20.409124    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh" event={"ID":"f0e70adf-7115-4e24-a2f7-e059aee572b2","Type":"ContainerStarted","Data":"1fad9ba053039f304205bf422c7869edc8799ab3280d01c84c9dc81debe32302"}
Oct 29 22:44:21.413508 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:21.413473    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh" event={"ID":"f0e70adf-7115-4e24-a2f7-e059aee572b2","Type":"ContainerStarted","Data":"08511cd0d90b0d24f2eee1e363c88a1d824e6ea97c002a8e3c1b62f8ec3e2a6f"}
Oct 29 22:44:21.492473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:21.492419    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-2scmh" podStartSLOduration=2.02856236 podStartE2EDuration="3.492407208s" totalImagesPullingTime="1.463844848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:21.492026597 +0000 UTC m=+118.595102868" watchObservedRunningTime="2026-10-29 22:44:21.492407208 +0000 UTC m=+118.595483479"
Oct 29 22:44:22.011762 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.011728    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:22.240581 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.240547    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:22.762213 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.762173    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"]
Oct 29 22:44:22.765371 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.765349    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.777095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.777069    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:22.804140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.804116    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:44:22.804140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.804134    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:44:22.804321 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.804307    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-dp659\""
Oct 29 22:44:22.804684 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.804667    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"]
Oct 29 22:44:22.875958 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.875928    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.876111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.875973    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.876208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.876112    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhxt\" (UniqueName: \"kubernetes.io/projected/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-kube-api-access-wwhxt\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.959157 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.959114    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:22.977283 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.977253    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.977439 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.977290    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.977439 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.977327    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhxt\" (UniqueName: \"kubernetes.io/projected/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-kube-api-access-wwhxt\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.977605 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.977586    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:22.977708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:22.977685    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:23.016906 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:23.016887    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:44:23.035006 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:23.034978    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:44:23.041553 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:23.041520    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhxt\" (UniqueName: \"kubernetes.io/projected/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-kube-api-access-wwhxt\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:23.095974 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:23.095949    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-dp659\""
Oct 29 22:44:23.100631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:23.100606    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:23.396615 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:23.396587    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"]
Oct 29 22:44:23.398182 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:44:23.398134    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17994aeb_b586_4ebe_b9b3_6bd0bf1657cc.slice/crio-6730dd8c3a68a109a8d667ca89dbf6c2cd706731d0e32ccbe06c9aab099c843c WatchSource:0}: Error finding container 6730dd8c3a68a109a8d667ca89dbf6c2cd706731d0e32ccbe06c9aab099c843c: Status 404 returned error can't find the container with id 6730dd8c3a68a109a8d667ca89dbf6c2cd706731d0e32ccbe06c9aab099c843c
Oct 29 22:44:23.419918 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:23.419892    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn" event={"ID":"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc","Type":"ContainerStarted","Data":"6730dd8c3a68a109a8d667ca89dbf6c2cd706731d0e32ccbe06c9aab099c843c"}
Oct 29 22:44:24.424243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:24.424141    4839 generic.go:356] "Generic (PLEG): container finished" podID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerID="0757bb868259b1fff78daeddd3d501219c76ef2af6853d09ef42350ad1de02d4" exitCode=0
Oct 29 22:44:24.424243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:24.424221    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn" event={"ID":"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc","Type":"ContainerDied","Data":"0757bb868259b1fff78daeddd3d501219c76ef2af6853d09ef42350ad1de02d4"}
Oct 29 22:44:25.150389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:25.150354    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:25.352904 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:25.352859    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:26.432752 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:26.432725    4839 generic.go:356] "Generic (PLEG): container finished" podID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerID="b76f27f11558d7a542bcf975f5eb1dfc91c89c4423ddebdad1cee2454ccf9246" exitCode=0
Oct 29 22:44:26.433197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:26.432759    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn" event={"ID":"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc","Type":"ContainerDied","Data":"b76f27f11558d7a542bcf975f5eb1dfc91c89c4423ddebdad1cee2454ccf9246"}
Oct 29 22:44:26.976662 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:26.976539    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:27.079680 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:27.079638    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:27.438262 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:27.438227    4839 generic.go:356] "Generic (PLEG): container finished" podID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerID="1f9423c13ef5beb5e9a2e71f427bcea7f7bf97a56ad754d3f5303c0e705a6340" exitCode=0
Oct 29 22:44:27.438649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:27.438287    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn" event={"ID":"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc","Type":"ContainerDied","Data":"1f9423c13ef5beb5e9a2e71f427bcea7f7bf97a56ad754d3f5303c0e705a6340"}
Oct 29 22:44:28.558879 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.558852    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:28.618560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.618533    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-kube-api-access-wwhxt\" (UniqueName: \"kubernetes.io/projected/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-kube-api-access-wwhxt\") pod \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") "
Oct 29 22:44:28.618707 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.618574    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-bundle\") pod \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") "
Oct 29 22:44:28.618707 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.618615    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-util\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-util\") pod \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\" (UID: \"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc\") "
Oct 29 22:44:28.619072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.619033    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-bundle" pod "17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" (UID: "17994aeb-b586-4ebe-b9b3-6bd0bf1657cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:28.620627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.620595    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-kube-api-access-wwhxt" pod "17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" (UID: "17994aeb-b586-4ebe-b9b3-6bd0bf1657cc"). InnerVolumeSpecName "kube-api-access-wwhxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:28.622900 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.622874    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-util" pod "17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" (UID: "17994aeb-b586-4ebe-b9b3-6bd0bf1657cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:28.719067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.718985    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhxt\" (UniqueName: \"kubernetes.io/projected/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-kube-api-access-wwhxt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:28.719067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.719015    4839 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-bundle\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:28.719067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:28.719025    4839 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17994aeb-b586-4ebe-b9b3-6bd0bf1657cc-util\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:29.446772 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:29.446741    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn"
Oct 29 22:44:29.446947 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:29.446741    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85lzmzn" event={"ID":"17994aeb-b586-4ebe-b9b3-6bd0bf1657cc","Type":"ContainerDied","Data":"6730dd8c3a68a109a8d667ca89dbf6c2cd706731d0e32ccbe06c9aab099c843c"}
Oct 29 22:44:29.446947 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:29.446866    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6730dd8c3a68a109a8d667ca89dbf6c2cd706731d0e32ccbe06c9aab099c843c"
Oct 29 22:44:30.597104 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.597069    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"]
Oct 29 22:44:30.599330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.597662    4839 cpu_manager.go:414] "containerMap: removing container" podUID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerName="util"
Oct 29 22:44:30.599330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.597701    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerName="util"
Oct 29 22:44:30.599330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.597726    4839 cpu_manager.go:414] "containerMap: removing container" podUID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerName="extract"
Oct 29 22:44:30.599330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.597736    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerName="extract"
Oct 29 22:44:30.599330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.597862    4839 cpu_manager.go:414] "containerMap: removing container" podUID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerName="pull"
Oct 29 22:44:30.599330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.597895    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerName="pull"
Oct 29 22:44:30.599330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.598120    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="17994aeb-b586-4ebe-b9b3-6bd0bf1657cc" containerName="extract"
Oct 29 22:44:30.603008 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.602982    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:30.636805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.636774    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 29 22:44:30.636970 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.636822    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 29 22:44:30.636970 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.636916    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5c4sc\""
Oct 29 22:44:30.637403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.637385    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"]
Oct 29 22:44:30.730589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.730557    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef38dee7-98b1-4e20-992d-240005ef83e1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-trqpq\" (UID: \"ef38dee7-98b1-4e20-992d-240005ef83e1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:30.730589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.730590    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhvgq\" (UniqueName: \"kubernetes.io/projected/ef38dee7-98b1-4e20-992d-240005ef83e1-kube-api-access-dhvgq\") pod \"cert-manager-webhook-5dbbbf75cc-trqpq\" (UID: \"ef38dee7-98b1-4e20-992d-240005ef83e1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:30.831890 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.831860    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef38dee7-98b1-4e20-992d-240005ef83e1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-trqpq\" (UID: \"ef38dee7-98b1-4e20-992d-240005ef83e1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:30.831996 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.831898    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhvgq\" (UniqueName: \"kubernetes.io/projected/ef38dee7-98b1-4e20-992d-240005ef83e1-kube-api-access-dhvgq\") pod \"cert-manager-webhook-5dbbbf75cc-trqpq\" (UID: \"ef38dee7-98b1-4e20-992d-240005ef83e1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:30.854537 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.854475    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef38dee7-98b1-4e20-992d-240005ef83e1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-trqpq\" (UID: \"ef38dee7-98b1-4e20-992d-240005ef83e1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:30.854628 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.854609    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhvgq\" (UniqueName: \"kubernetes.io/projected/ef38dee7-98b1-4e20-992d-240005ef83e1-kube-api-access-dhvgq\") pod \"cert-manager-webhook-5dbbbf75cc-trqpq\" (UID: \"ef38dee7-98b1-4e20-992d-240005ef83e1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:30.927736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:30.927704    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:31.218677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:31.218653    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"]
Oct 29 22:44:31.220385 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:44:31.220351    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef38dee7_98b1_4e20_992d_240005ef83e1.slice/crio-ebb31c6d485c16e3194abcfe6c6cf05db3427236f7c3fc0e4ae5bf6248f392c4 WatchSource:0}: Error finding container ebb31c6d485c16e3194abcfe6c6cf05db3427236f7c3fc0e4ae5bf6248f392c4: Status 404 returned error can't find the container with id ebb31c6d485c16e3194abcfe6c6cf05db3427236f7c3fc0e4ae5bf6248f392c4
Oct 29 22:44:31.454156 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:31.454121    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq" event={"ID":"ef38dee7-98b1-4e20-992d-240005ef83e1","Type":"ContainerStarted","Data":"ebb31c6d485c16e3194abcfe6c6cf05db3427236f7c3fc0e4ae5bf6248f392c4"}
Oct 29 22:44:32.937558 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:32.937525    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:33.239357 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:44:33.239280    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:44:33.852322 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:33.852242    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:33.980168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:33.980117    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:34.465525 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:34.465481    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq" event={"ID":"ef38dee7-98b1-4e20-992d-240005ef83e1","Type":"ContainerStarted","Data":"ebcccfac3b8bc2671703e658b76446f1aa935e4d18a38ba27d7af2fda5dd05b4"}
Oct 29 22:44:34.465697 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:34.465538    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:34.507562 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:34.507509    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq" podStartSLOduration=2.165876879 podStartE2EDuration="4.507497107s" totalImagesPullingTime="2.341620228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:34.506997185 +0000 UTC m=+131.610073458" watchObservedRunningTime="2026-10-29 22:44:34.507497107 +0000 UTC m=+131.610573375"
Oct 29 22:44:37.678065 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:37.677990    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:37.818873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:37.818834    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:39.780345 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.780313    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"]
Oct 29 22:44:39.843902 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.843872    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"]
Oct 29 22:44:39.844051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.843965    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:39.874783 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.874758    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 29 22:44:39.874900 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.874809    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cszxf\""
Oct 29 22:44:39.874900 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.874834    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 29 22:44:39.895432 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.895405    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxwb\" (UniqueName: \"kubernetes.io/projected/37021761-19c5-40e9-87e5-4121882dcb17-kube-api-access-9fxwb\") pod \"openshift-lws-operator-8465994787-pxd99\" (UID: \"37021761-19c5-40e9-87e5-4121882dcb17\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:39.895557 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.895439    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37021761-19c5-40e9-87e5-4121882dcb17-tmp\") pod \"openshift-lws-operator-8465994787-pxd99\" (UID: \"37021761-19c5-40e9-87e5-4121882dcb17\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:39.996377 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.996341    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxwb\" (UniqueName: \"kubernetes.io/projected/37021761-19c5-40e9-87e5-4121882dcb17-kube-api-access-9fxwb\") pod \"openshift-lws-operator-8465994787-pxd99\" (UID: \"37021761-19c5-40e9-87e5-4121882dcb17\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:39.996505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.996396    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37021761-19c5-40e9-87e5-4121882dcb17-tmp\") pod \"openshift-lws-operator-8465994787-pxd99\" (UID: \"37021761-19c5-40e9-87e5-4121882dcb17\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:39.996796 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:39.996778    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/37021761-19c5-40e9-87e5-4121882dcb17-tmp\") pod \"openshift-lws-operator-8465994787-pxd99\" (UID: \"37021761-19c5-40e9-87e5-4121882dcb17\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:40.017811 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:40.017787    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxwb\" (UniqueName: \"kubernetes.io/projected/37021761-19c5-40e9-87e5-4121882dcb17-kube-api-access-9fxwb\") pod \"openshift-lws-operator-8465994787-pxd99\" (UID: \"37021761-19c5-40e9-87e5-4121882dcb17\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:40.167697 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:40.167617    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"
Oct 29 22:44:40.471461 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:40.471386    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq"
Oct 29 22:44:40.476421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:40.476402    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-pxd99"]
Oct 29 22:44:40.477651 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:44:40.477626    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37021761_19c5_40e9_87e5_4121882dcb17.slice/crio-96f490ab73b48d4fc2dc1a3dfe7a0dc8cd4786d554152aa954549bc4ae9536aa WatchSource:0}: Error finding container 96f490ab73b48d4fc2dc1a3dfe7a0dc8cd4786d554152aa954549bc4ae9536aa: Status 404 returned error can't find the container with id 96f490ab73b48d4fc2dc1a3dfe7a0dc8cd4786d554152aa954549bc4ae9536aa
Oct 29 22:44:40.487833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:40.487796    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99" event={"ID":"37021761-19c5-40e9-87e5-4121882dcb17","Type":"ContainerStarted","Data":"96f490ab73b48d4fc2dc1a3dfe7a0dc8cd4786d554152aa954549bc4ae9536aa"}
Oct 29 22:44:42.496684 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:42.496648    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99" event={"ID":"37021761-19c5-40e9-87e5-4121882dcb17","Type":"ContainerStarted","Data":"62d0c028ccd89427807c782a27d3850c742e1ccb737460929b348649d6436224"}
Oct 29 22:44:42.541819 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:42.541696    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-pxd99" podStartSLOduration=1.751582008 podStartE2EDuration="3.541676727s" totalImagesPullingTime="1.790094719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:44:42.540841336 +0000 UTC m=+139.643917611" watchObservedRunningTime="2026-10-29 22:44:42.541676727 +0000 UTC m=+139.644752999"
Oct 29 22:44:43.562374 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:43.562328    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:43.662854 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:43.662824    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:43.819333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:43.819239    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:45.477213 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:45.477171    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:46.958452 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:46.958420    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"]
Oct 29 22:44:46.961731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:46.961713    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:46.993968 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:46.993934    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:44:46.994117 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:46.994024    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:44:46.994117 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:46.994071    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-xlx2j\""
Oct 29 22:44:46.994696 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:46.994679    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"]
Oct 29 22:44:47.043870 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.043833    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.044043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.043880    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.044043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.043905    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gtf\" (UniqueName: \"kubernetes.io/projected/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-kube-api-access-v5gtf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.144343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.144308    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.144504 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.144357    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.144504 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.144378    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gtf\" (UniqueName: \"kubernetes.io/projected/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-kube-api-access-v5gtf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.144672 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.144653    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.144713 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.144686    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.165771 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.165733    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gtf\" (UniqueName: \"kubernetes.io/projected/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-kube-api-access-v5gtf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.276915 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.276888    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:47.567757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:47.567733    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"]
Oct 29 22:44:47.569361 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:44:47.569331    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d2f0d4_b5d4_4f53_b1fc_c2a98a84f4af.slice/crio-a7cfa60d72bee8e03f8e712d32bee13cdc67dbce6c0e4e4117c41b62b02f6e7c WatchSource:0}: Error finding container a7cfa60d72bee8e03f8e712d32bee13cdc67dbce6c0e4e4117c41b62b02f6e7c: Status 404 returned error can't find the container with id a7cfa60d72bee8e03f8e712d32bee13cdc67dbce6c0e4e4117c41b62b02f6e7c
Oct 29 22:44:48.517969 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:48.517932    4839 generic.go:356] "Generic (PLEG): container finished" podID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerID="cf06f2ba4d26424d7c4a5a1235efe0903871324f35d6ec7a0ee5bff625066182" exitCode=0
Oct 29 22:44:48.518368 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:48.517986    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f" event={"ID":"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af","Type":"ContainerDied","Data":"cf06f2ba4d26424d7c4a5a1235efe0903871324f35d6ec7a0ee5bff625066182"}
Oct 29 22:44:48.518368 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:48.518065    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f" event={"ID":"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af","Type":"ContainerStarted","Data":"a7cfa60d72bee8e03f8e712d32bee13cdc67dbce6c0e4e4117c41b62b02f6e7c"}
Oct 29 22:44:49.523772 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:49.523734    4839 generic.go:356] "Generic (PLEG): container finished" podID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerID="9b3da708be32db9a31345aab465c8ca25fe1ead09117cbaad2211978e4ae9c44" exitCode=0
Oct 29 22:44:49.524282 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:49.523816    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f" event={"ID":"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af","Type":"ContainerDied","Data":"9b3da708be32db9a31345aab465c8ca25fe1ead09117cbaad2211978e4ae9c44"}
Oct 29 22:44:50.529330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:50.529284    4839 generic.go:356] "Generic (PLEG): container finished" podID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerID="1116c0a525c19e82594678f354c582e0bfa16b72a3bf0865222307ab3c5bcc65" exitCode=0
Oct 29 22:44:50.529719 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:50.529363    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f" event={"ID":"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af","Type":"ContainerDied","Data":"1116c0a525c19e82594678f354c582e0bfa16b72a3bf0865222307ab3c5bcc65"}
Oct 29 22:44:51.659569 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.659542    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:51.778398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.778366    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-kube-api-access-v5gtf\" (UniqueName: \"kubernetes.io/projected/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-kube-api-access-v5gtf\") pod \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") "
Oct 29 22:44:51.778398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.778405    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-bundle\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-bundle\") pod \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") "
Oct 29 22:44:51.778602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.778428    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-util\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-util\") pod \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\" (UID: \"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af\") "
Oct 29 22:44:51.778994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.778953    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-bundle" pod "94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" (UID: "94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:51.780427 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.780406    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-kube-api-access-v5gtf" pod "94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" (UID: "94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af"). InnerVolumeSpecName "kube-api-access-v5gtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:44:51.784160 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.784122    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-util" pod "94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" (UID: "94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:44:51.879872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.879836    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gtf\" (UniqueName: \"kubernetes.io/projected/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-kube-api-access-v5gtf\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:51.879872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.879866    4839 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-bundle\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:51.879872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:51.879874    4839 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af-util\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:44:52.538010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:52.537968    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f" event={"ID":"94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af","Type":"ContainerDied","Data":"a7cfa60d72bee8e03f8e712d32bee13cdc67dbce6c0e4e4117c41b62b02f6e7c"}
Oct 29 22:44:52.538010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:52.538005    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ggw5f"
Oct 29 22:44:52.538010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:52.538018    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7cfa60d72bee8e03f8e712d32bee13cdc67dbce6c0e4e4117c41b62b02f6e7c"
Oct 29 22:44:57.012564 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.012530    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:57.135159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.135126    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:57.576311 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.576273    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:57.755086 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.755056    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:57.812883 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.812849    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"]
Oct 29 22:44:57.813080 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.813068    4839 cpu_manager.go:414] "containerMap: removing container" podUID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerName="extract"
Oct 29 22:44:57.813138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.813081    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerName="extract"
Oct 29 22:44:57.813138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.813097    4839 cpu_manager.go:414] "containerMap: removing container" podUID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerName="pull"
Oct 29 22:44:57.813138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.813102    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerName="pull"
Oct 29 22:44:57.813138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.813117    4839 cpu_manager.go:414] "containerMap: removing container" podUID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerName="util"
Oct 29 22:44:57.813138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.813122    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerName="util"
Oct 29 22:44:57.813334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.813172    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d2f0d4-b5d4-4f53-b1fc-c2a98a84f4af" containerName="extract"
Oct 29 22:44:57.816421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.816401    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:57.864589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.860368    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"]
Oct 29 22:44:57.898718 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.898679    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 29 22:44:57.898718 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.898709    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 29 22:44:57.898914 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.898732    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 29 22:44:57.898914 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.898781    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k79nx\""
Oct 29 22:44:57.900560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.900537    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 29 22:44:57.900630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.900587    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 29 22:44:57.918028 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.917999    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-metrics-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:57.918140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.918033    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19f65fc3-d04a-4a71-b3d2-9a27f001912c-tmp\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:57.918140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.918061    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19f65fc3-d04a-4a71-b3d2-9a27f001912c-manifests\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:57.918140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.918085    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66s7\" (UniqueName: \"kubernetes.io/projected/19f65fc3-d04a-4a71-b3d2-9a27f001912c-kube-api-access-h66s7\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:57.918318 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.918198    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:57.918318 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.918237    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-webhook-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:57.965292 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:57.965262    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:58.019078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.019040    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-metrics-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.019078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.019077    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19f65fc3-d04a-4a71-b3d2-9a27f001912c-tmp\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.019621 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.019102    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19f65fc3-d04a-4a71-b3d2-9a27f001912c-manifests\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.019621 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.019128    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66s7\" (UniqueName: \"kubernetes.io/projected/19f65fc3-d04a-4a71-b3d2-9a27f001912c-kube-api-access-h66s7\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.019621 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.019157    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.019621 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.019172    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-webhook-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.019792 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.019677    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/19f65fc3-d04a-4a71-b3d2-9a27f001912c-manifests\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.021501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.021473    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19f65fc3-d04a-4a71-b3d2-9a27f001912c-tmp\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.021746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.021726    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-webhook-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.021897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.021879    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-metrics-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.021935 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.021914    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/19f65fc3-d04a-4a71-b3d2-9a27f001912c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.052468 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.052441    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66s7\" (UniqueName: \"kubernetes.io/projected/19f65fc3-d04a-4a71-b3d2-9a27f001912c-kube-api-access-h66s7\") pod \"opendatahub-operator-controller-manager-675b698459-ctkl2\" (UID: \"19f65fc3-d04a-4a71-b3d2-9a27f001912c\") " pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.075773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.075742    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:58.130962 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.130905    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:44:58.184575 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.184542    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:44:58.481802 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.481777    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"]
Oct 29 22:44:58.483076 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:44:58.483045    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f65fc3_d04a_4a71_b3d2_9a27f001912c.slice/crio-abc6df706839041814e5ddcb0fd8682e8b79e6236b899107df31ac6e0fc17a8b WatchSource:0}: Error finding container abc6df706839041814e5ddcb0fd8682e8b79e6236b899107df31ac6e0fc17a8b: Status 404 returned error can't find the container with id abc6df706839041814e5ddcb0fd8682e8b79e6236b899107df31ac6e0fc17a8b
Oct 29 22:44:58.560130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:44:58.560099    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2" event={"ID":"19f65fc3-d04a-4a71-b3d2-9a27f001912c","Type":"ContainerStarted","Data":"abc6df706839041814e5ddcb0fd8682e8b79e6236b899107df31ac6e0fc17a8b"}
Oct 29 22:45:00.284514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:00.284478    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:00.401497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:00.401468    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:01.574411 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:01.574321    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2" event={"ID":"19f65fc3-d04a-4a71-b3d2-9a27f001912c","Type":"ContainerStarted","Data":"1717364be482fcc644471d8bbc7357d83068a59e7cac9917869d0ac085159228"}
Oct 29 22:45:01.973422 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:01.973326    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:02.102795 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:02.102752    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:02.342371 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:02.342315    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:02.579758 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:02.579723    4839 generic.go:356] "Generic (PLEG): container finished" podID="19f65fc3-d04a-4a71-b3d2-9a27f001912c" containerID="1717364be482fcc644471d8bbc7357d83068a59e7cac9917869d0ac085159228" exitCode=0
Oct 29 22:45:02.579758 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:02.579759    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2" event={"ID":"19f65fc3-d04a-4a71-b3d2-9a27f001912c","Type":"ContainerDied","Data":"1717364be482fcc644471d8bbc7357d83068a59e7cac9917869d0ac085159228"}
Oct 29 22:45:03.237298 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:45:03.237211    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:45:03.588034 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:03.587984    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2" event={"ID":"19f65fc3-d04a-4a71-b3d2-9a27f001912c","Type":"ContainerStarted","Data":"35890de1ec939994175a79d59442de73d036d662234006e04885e72c05246565"}
Oct 29 22:45:03.588560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:03.588139    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:45:13.553883 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:13.553847    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:14.594159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:14.594116    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2"
Oct 29 22:45:14.696335 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:14.696274    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-675b698459-ctkl2" podStartSLOduration=14.08956327 podStartE2EDuration="17.696260663s" totalImagesPullingTime="3.170909632s" totalInitContainerRuntime="435.787761ms" isStatefulPod=false podCreationTimestamp="2026-10-29 22:44:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:03.694039849 +0000 UTC m=+160.797116121" watchObservedRunningTime="2026-10-29 22:45:14.696260663 +0000 UTC m=+171.799336933"
Oct 29 22:45:17.227021 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.226980    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:17.340723 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.340686    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:17.680998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.680965    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"]
Oct 29 22:45:17.687567 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.687541    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.688423 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.688398    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:17.719132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.719103    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 29 22:45:17.719294 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.719204    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 29 22:45:17.719380 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.719361    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 29 22:45:17.719433 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.719422    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 29 22:45:17.719505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.719489    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6fhn4\""
Oct 29 22:45:17.720340 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.720314    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"]
Oct 29 22:45:17.839764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.839734    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:17.862382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.862354    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8469ab-1187-4d92-9399-863ea18599f9-tls-certs\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.862514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.862385    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8469ab-1187-4d92-9399-863ea18599f9-tmp\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.862514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.862416    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5g4\" (UniqueName: \"kubernetes.io/projected/8f8469ab-1187-4d92-9399-863ea18599f9-kube-api-access-qq5g4\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.963078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.962989    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8469ab-1187-4d92-9399-863ea18599f9-tls-certs\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.963078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.963032    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8469ab-1187-4d92-9399-863ea18599f9-tmp\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.963078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.963067    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5g4\" (UniqueName: \"kubernetes.io/projected/8f8469ab-1187-4d92-9399-863ea18599f9-kube-api-access-qq5g4\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.965347 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.965311    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f8469ab-1187-4d92-9399-863ea18599f9-tmp\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.965574 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.965552    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f8469ab-1187-4d92-9399-863ea18599f9-tls-certs\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:17.985416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:17.985381    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5g4\" (UniqueName: \"kubernetes.io/projected/8f8469ab-1187-4d92-9399-863ea18599f9-kube-api-access-qq5g4\") pod \"kube-auth-proxy-84d6d655bb-nssk5\" (UID: \"8f8469ab-1187-4d92-9399-863ea18599f9\") " pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:18.000082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.000056    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"
Oct 29 22:45:18.295830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.295796    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5"]
Oct 29 22:45:18.296825 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:45:18.296801    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8469ab_1187_4d92_9399_863ea18599f9.slice/crio-d27cbf3fa878208ac78e3895e2ee7e077171dde021caf5dd8f96b97142caade4 WatchSource:0}: Error finding container d27cbf3fa878208ac78e3895e2ee7e077171dde021caf5dd8f96b97142caade4: Status 404 returned error can't find the container with id d27cbf3fa878208ac78e3895e2ee7e077171dde021caf5dd8f96b97142caade4
Oct 29 22:45:18.644549 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.644467    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5" event={"ID":"8f8469ab-1187-4d92-9399-863ea18599f9","Type":"ContainerStarted","Data":"d27cbf3fa878208ac78e3895e2ee7e077171dde021caf5dd8f96b97142caade4"}
Oct 29 22:45:18.856961 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.856926    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"]
Oct 29 22:45:18.862130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.862106    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:18.882005 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.881976    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"]
Oct 29 22:45:18.894978 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.894908    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 29 22:45:18.906249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.906226    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cnhp8\""
Oct 29 22:45:18.906249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.906243    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 29 22:45:18.906412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.906235    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 29 22:45:18.970057 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.970021    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-metrics-cert\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:18.970251 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.970069    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-cert\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:18.970251 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.970100    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-manager-config\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:18.970355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:18.970268    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f948n\" (UniqueName: \"kubernetes.io/projected/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-kube-api-access-f948n\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.071300 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.071247    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f948n\" (UniqueName: \"kubernetes.io/projected/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-kube-api-access-f948n\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.071482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.071322    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-metrics-cert\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.071482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.071358    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-cert\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.071482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.071418    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-manager-config\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.072690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.072659    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-manager-config\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.075695 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.075673    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-metrics-cert\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.075816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.075753    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-cert\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.099014 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.098979    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f948n\" (UniqueName: \"kubernetes.io/projected/a549fd5f-8993-4fcb-b71f-38ed48d5b4dd-kube-api-access-f948n\") pod \"lws-controller-manager-59b8cf9b84-8mdq7\" (UID: \"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd\") " pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.174214 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.173931    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:19.213830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.213796    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:19.327929 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.327878    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:19.487311 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.487286    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"]
Oct 29 22:45:19.489019 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:45:19.488989    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda549fd5f_8993_4fcb_b71f_38ed48d5b4dd.slice/crio-7eab79a2c4dc2bbaf3c963b7fa75bc70093169a709012869c3f4efb54b6e35a2 WatchSource:0}: Error finding container 7eab79a2c4dc2bbaf3c963b7fa75bc70093169a709012869c3f4efb54b6e35a2: Status 404 returned error can't find the container with id 7eab79a2c4dc2bbaf3c963b7fa75bc70093169a709012869c3f4efb54b6e35a2
Oct 29 22:45:19.650948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:19.650913    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7" event={"ID":"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd","Type":"ContainerStarted","Data":"7eab79a2c4dc2bbaf3c963b7fa75bc70093169a709012869c3f4efb54b6e35a2"}
Oct 29 22:45:20.657978 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:20.657930    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:20.870199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:20.870086    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:21.138633 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.138596    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-n78vt"]
Oct 29 22:45:21.141864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.141840    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:21.158979 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.158953    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\""
Oct 29 22:45:21.173333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.173299    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-ddk5s\""
Oct 29 22:45:21.174283 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.174261    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-n78vt"]
Oct 29 22:45:21.189199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.189173    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:21.288067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.288031    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwdh\" (UniqueName: \"kubernetes.io/projected/91cd164a-636c-4878-b004-af3bd0eda83d-kube-api-access-5hwdh\") pod \"odh-observability-584f99c788-n78vt\" (UID: \"91cd164a-636c-4878-b004-af3bd0eda83d\") " pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:21.288268 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.288083    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/91cd164a-636c-4878-b004-af3bd0eda83d-webhook-certs\") pod \"odh-observability-584f99c788-n78vt\" (UID: \"91cd164a-636c-4878-b004-af3bd0eda83d\") " pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:21.388965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.388926    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/91cd164a-636c-4878-b004-af3bd0eda83d-webhook-certs\") pod \"odh-observability-584f99c788-n78vt\" (UID: \"91cd164a-636c-4878-b004-af3bd0eda83d\") " pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:21.389188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.388985    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwdh\" (UniqueName: \"kubernetes.io/projected/91cd164a-636c-4878-b004-af3bd0eda83d-kube-api-access-5hwdh\") pod \"odh-observability-584f99c788-n78vt\" (UID: \"91cd164a-636c-4878-b004-af3bd0eda83d\") " pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:21.389188 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:45:21.389120    4839 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found
Oct 29 22:45:21.389352 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:45:21.389236    4839 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91cd164a-636c-4878-b004-af3bd0eda83d-webhook-certs podName:91cd164a-636c-4878-b004-af3bd0eda83d nodeName:}" failed. No retries permitted until 2026-10-29 22:45:21.889212237 +0000 UTC m=+178.992288502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/91cd164a-636c-4878-b004-af3bd0eda83d-webhook-certs") pod "odh-observability-584f99c788-n78vt" (UID: "91cd164a-636c-4878-b004-af3bd0eda83d") : secret "odh-observability-webhook-cert" not found
Oct 29 22:45:21.410583 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.410548    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwdh\" (UniqueName: \"kubernetes.io/projected/91cd164a-636c-4878-b004-af3bd0eda83d-kube-api-access-5hwdh\") pod \"odh-observability-584f99c788-n78vt\" (UID: \"91cd164a-636c-4878-b004-af3bd0eda83d\") " pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:21.661417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.661291    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5" event={"ID":"8f8469ab-1187-4d92-9399-863ea18599f9","Type":"ContainerStarted","Data":"24fcf5974e31abeb753eca20479d6a44b67ebd046d9a3a22ceda407a7d003074"}
Oct 29 22:45:21.662932 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.662902    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7" event={"ID":"a549fd5f-8993-4fcb-b71f-38ed48d5b4dd","Type":"ContainerStarted","Data":"f8bd40da33cddb469223b2e6509d072b4a6159e9942641f1350aa80d242ced8f"}
Oct 29 22:45:21.663076 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.662994    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:21.663210 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.663194    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:21.704872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.704821    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-84d6d655bb-nssk5" podStartSLOduration=1.990151636 podStartE2EDuration="4.704807349s" totalImagesPullingTime="2.714655713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:21.703929881 +0000 UTC m=+178.807006151" watchObservedRunningTime="2026-10-29 22:45:21.704807349 +0000 UTC m=+178.807883620"
Oct 29 22:45:21.747313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.747246    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7" podStartSLOduration=2.232525343 podStartE2EDuration="3.747231212s" totalImagesPullingTime="1.514705869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:21.746900447 +0000 UTC m=+178.849976718" watchObservedRunningTime="2026-10-29 22:45:21.747231212 +0000 UTC m=+178.850307485"
Oct 29 22:45:21.893170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.893117    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/91cd164a-636c-4878-b004-af3bd0eda83d-webhook-certs\") pod \"odh-observability-584f99c788-n78vt\" (UID: \"91cd164a-636c-4878-b004-af3bd0eda83d\") " pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:21.895720 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:21.895692    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/91cd164a-636c-4878-b004-af3bd0eda83d-webhook-certs\") pod \"odh-observability-584f99c788-n78vt\" (UID: \"91cd164a-636c-4878-b004-af3bd0eda83d\") " pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:22.053943 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:22.053902    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:22.346842 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:22.346816    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-n78vt"]
Oct 29 22:45:22.348058 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:45:22.348030    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91cd164a_636c_4878_b004_af3bd0eda83d.slice/crio-8f4d79b8efe904d1426e30e7e8a0bfa9f6ff40a93d53738ed24be8f2c5010a9e WatchSource:0}: Error finding container 8f4d79b8efe904d1426e30e7e8a0bfa9f6ff40a93d53738ed24be8f2c5010a9e: Status 404 returned error can't find the container with id 8f4d79b8efe904d1426e30e7e8a0bfa9f6ff40a93d53738ed24be8f2c5010a9e
Oct 29 22:45:22.667848 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:22.667769    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-n78vt" event={"ID":"91cd164a-636c-4878-b004-af3bd0eda83d","Type":"ContainerStarted","Data":"8f4d79b8efe904d1426e30e7e8a0bfa9f6ff40a93d53738ed24be8f2c5010a9e"}
Oct 29 22:45:22.802632 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:22.802596    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:22.918473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:22.918406    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:23.592003 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:23.591973    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:23.691670 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:23.691632    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:23.856526 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:23.856444    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:24.682967 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:24.682934    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-n78vt" event={"ID":"91cd164a-636c-4878-b004-af3bd0eda83d","Type":"ContainerStarted","Data":"e7487fa5252e0b334c6e70dab0c3dd3c31e25410294fc3b4626f7ed21e27bf06"}
Oct 29 22:45:24.683194 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:24.683049    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:45:24.726223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:24.726172    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-n78vt" podStartSLOduration=1.4876006880000001 podStartE2EDuration="3.7261404s" totalImagesPullingTime="2.238539712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:45:24.725921323 +0000 UTC m=+181.828997593" watchObservedRunningTime="2026-10-29 22:45:24.7261404 +0000 UTC m=+181.829216670"
Oct 29 22:45:32.670292 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:32.670253    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59b8cf9b84-8mdq7"
Oct 29 22:45:33.236388 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:45:33.236349    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:45:33.737337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:33.737293    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:33.909798 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:33.909770    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:45:35.689236 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:45:35.689207    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-n78vt"
Oct 29 22:46:03.236609 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:46:03.236575    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:46:07.698677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:07.698593    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:07.799528 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:07.799495    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:12.173554 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:12.173522    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:12.335610 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:12.335575    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:13.669965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:13.669933    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:18.933057 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.933009    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"]
Oct 29 22:46:18.936158 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.936131    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:18.960110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.960079    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 29 22:46:18.960365 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.960344    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 29 22:46:18.960501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.960377    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 29 22:46:18.960501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.960396    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 29 22:46:18.960501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.960353    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wdpvc\""
Oct 29 22:46:18.975573 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.975544    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"]
Oct 29 22:46:18.997105 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.997061    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcjw\" (UniqueName: \"kubernetes.io/projected/0660ba50-ece7-4b6d-a7e1-223cd5a72603-kube-api-access-8jcjw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:18.997105 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.997105    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:18.997355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.997204    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:18.997355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.997243    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:18.997355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.997297    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:18.997355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.997343    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:18.997553 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:18.997364    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0660ba50-ece7-4b6d-a7e1-223cd5a72603-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.098501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.098460    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0660ba50-ece7-4b6d-a7e1-223cd5a72603-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.098690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.098514    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcjw\" (UniqueName: \"kubernetes.io/projected/0660ba50-ece7-4b6d-a7e1-223cd5a72603-kube-api-access-8jcjw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.098690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.098542    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.098690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.098575    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.098690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.098610    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.098690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.098639    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.098690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.098671    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.099539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.099509    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.101098 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.101074    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0660ba50-ece7-4b6d-a7e1-223cd5a72603-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.101201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.101091    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.101201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.101107    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.101201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.101099    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.120812 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.120784    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcjw\" (UniqueName: \"kubernetes.io/projected/0660ba50-ece7-4b6d-a7e1-223cd5a72603-kube-api-access-8jcjw\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.123139 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.123115    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0660ba50-ece7-4b6d-a7e1-223cd5a72603-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lbn4t\" (UID: \"0660ba50-ece7-4b6d-a7e1-223cd5a72603\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.247002 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.246914    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:19.558613 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.558578    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"]
Oct 29 22:46:19.559807 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:46:19.559780    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0660ba50_ece7_4b6d_a7e1_223cd5a72603.slice/crio-795d685f0e126df0bc261c7ad115230ba68d06eaebe8b144ca3aacfb3fbb5d51 WatchSource:0}: Error finding container 795d685f0e126df0bc261c7ad115230ba68d06eaebe8b144ca3aacfb3fbb5d51: Status 404 returned error can't find the container with id 795d685f0e126df0bc261c7ad115230ba68d06eaebe8b144ca3aacfb3fbb5d51
Oct 29 22:46:19.891505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.891410    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t" event={"ID":"0660ba50-ece7-4b6d-a7e1-223cd5a72603","Type":"ContainerStarted","Data":"795d685f0e126df0bc261c7ad115230ba68d06eaebe8b144ca3aacfb3fbb5d51"}
Oct 29 22:46:19.952125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:19.952094    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.051096 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:20.051060    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.170053 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:20.169972    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.334495 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:20.334458    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.480246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:20.480134    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:20.797271 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:20.797233    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.461602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.461573    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:46:22.461880 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.461633    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:46:22.594998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.594970    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.696710 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.696672    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.797773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.797740    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.903921 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.903884    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:22.906102 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.906074    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t" event={"ID":"0660ba50-ece7-4b6d-a7e1-223cd5a72603","Type":"ContainerStarted","Data":"426dfe8f3ee9d3971e46f09be7a1e40f4963fb23ebc999f8494eeb5f4d3d58d5"}
Oct 29 22:46:22.906263 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:22.906187    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:23.035880 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:23.035827    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:23.912077 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:23.912043    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t"
Oct 29 22:46:23.920707 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:23.920676    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:46:23.989211 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:23.989140    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lbn4t" podStartSLOduration=3.090025547 podStartE2EDuration="5.98912552s" totalImagesPullingTime="2.899099973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:46:22.989470144 +0000 UTC m=+240.092546414" watchObservedRunningTime="2026-10-29 22:46:23.98912552 +0000 UTC m=+241.092201790"
Oct 29 22:46:33.237512 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:46:33.237473    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:46:44.492801 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:46:44.492764    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:03.237328 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:47:03.237279    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:47:10.626916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:10.626862    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:10.786798 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:10.786757    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:12.835432 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.835396    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"]
Oct 29 22:47:12.838724 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.838705    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.870066 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.870034    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-7rm5b\""
Oct 29 22:47:12.871032 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.871008    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"]
Oct 29 22:47:12.871728 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.871713    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 29 22:47:12.871785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.871717    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 29 22:47:12.892119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.892084    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.892275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.892123    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.892275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.892180    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qv8w\" (UniqueName: \"kubernetes.io/projected/5cdbf12a-610c-4c00-b552-b80bafafa93b-kube-api-access-4qv8w\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.943890 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.943850    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"]
Oct 29 22:47:12.947281 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.947256    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:12.971830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.971803    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-fjcwc\""
Oct 29 22:47:12.973464 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.973433    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"]
Oct 29 22:47:12.992571 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.992542    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcw4v\" (UniqueName: \"kubernetes.io/projected/dd110d23-f5ae-4543-b782-7420a434db46-kube-api-access-vcw4v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:12.992700 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.992582    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:12.992742 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.992696    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:12.992783 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.992741    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.992783 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.992763    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.992873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.992795    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qv8w\" (UniqueName: \"kubernetes.io/projected/5cdbf12a-610c-4c00-b552-b80bafafa93b-kube-api-access-4qv8w\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.993092 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.993067    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:12.993195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:12.993107    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:13.018096 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.018063    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qv8w\" (UniqueName: \"kubernetes.io/projected/5cdbf12a-610c-4c00-b552-b80bafafa93b-kube-api-access-4qv8w\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:13.037674 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.037644    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"]
Oct 29 22:47:13.041186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.041166    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.061392 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.061363    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"]
Oct 29 22:47:13.063195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.063140    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8lhzq\""
Oct 29 22:47:13.093638 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093556    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:13.093638 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093604    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpgqn\" (UniqueName: \"kubernetes.io/projected/340767b7-0ee1-4dab-96c0-e31c16a4cc91-kube-api-access-xpgqn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.093638 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093637    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:13.093858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093660    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.093858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093694    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.093858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093808    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcw4v\" (UniqueName: \"kubernetes.io/projected/dd110d23-f5ae-4543-b782-7420a434db46-kube-api-access-vcw4v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:13.094003 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093983    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:13.094057 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.093998    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:13.116444 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.116414    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcw4v\" (UniqueName: \"kubernetes.io/projected/dd110d23-f5ae-4543-b782-7420a434db46-kube-api-access-vcw4v\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:13.140042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.140012    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"]
Oct 29 22:47:13.143465 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.143444    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.153957 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.153933    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:13.164736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.164706    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-5sl6q\""
Oct 29 22:47:13.167444 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.167418    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"]
Oct 29 22:47:13.194474 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194437    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.194652 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194497    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.194652 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194529    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjvxh\" (UniqueName: \"kubernetes.io/projected/73e82de8-f42b-43de-982c-ff96e03970fd-kube-api-access-fjvxh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.194652 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194575    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpgqn\" (UniqueName: \"kubernetes.io/projected/340767b7-0ee1-4dab-96c0-e31c16a4cc91-kube-api-access-xpgqn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.194815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194729    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.194815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194778    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.194956 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194838    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.194956 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.194888    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.217122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.217091    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpgqn\" (UniqueName: \"kubernetes.io/projected/340767b7-0ee1-4dab-96c0-e31c16a4cc91-kube-api-access-xpgqn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.263042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.263009    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:13.295540 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.295511    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.295661 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.295545    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.295661 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.295588    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjvxh\" (UniqueName: \"kubernetes.io/projected/73e82de8-f42b-43de-982c-ff96e03970fd-kube-api-access-fjvxh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.295917 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.295895    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.296026 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.296004    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.321615 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.321589    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjvxh\" (UniqueName: \"kubernetes.io/projected/73e82de8-f42b-43de-982c-ff96e03970fd-kube-api-access-fjvxh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.356810 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.356741    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:13.464175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.464132    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"]
Oct 29 22:47:13.465464 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:47:13.465438    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdbf12a_610c_4c00_b552_b80bafafa93b.slice/crio-895e9fac68cfe6f2173f3930c92791f15c7def28bb6575607235e447464c8b60 WatchSource:0}: Error finding container 895e9fac68cfe6f2173f3930c92791f15c7def28bb6575607235e447464c8b60: Status 404 returned error can't find the container with id 895e9fac68cfe6f2173f3930c92791f15c7def28bb6575607235e447464c8b60
Oct 29 22:47:13.468588 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.468564    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:13.562138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.562112    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"]
Oct 29 22:47:13.589985 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:47:13.589947    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd110d23_f5ae_4543_b782_7420a434db46.slice/crio-87b7a0f2dabd1d4a003df77a76b4b66b1958f772f36c40e72bbaa1529d6e83ed WatchSource:0}: Error finding container 87b7a0f2dabd1d4a003df77a76b4b66b1958f772f36c40e72bbaa1529d6e83ed: Status 404 returned error can't find the container with id 87b7a0f2dabd1d4a003df77a76b4b66b1958f772f36c40e72bbaa1529d6e83ed
Oct 29 22:47:13.649817 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.649790    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"]
Oct 29 22:47:13.650351 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:47:13.650321    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340767b7_0ee1_4dab_96c0_e31c16a4cc91.slice/crio-0864e0faa79fd880a71b15acbcea0341e276caf6f33e5259963d7827fa75ae03 WatchSource:0}: Error finding container 0864e0faa79fd880a71b15acbcea0341e276caf6f33e5259963d7827fa75ae03: Status 404 returned error can't find the container with id 0864e0faa79fd880a71b15acbcea0341e276caf6f33e5259963d7827fa75ae03
Oct 29 22:47:13.764247 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.764225    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"]
Oct 29 22:47:13.785897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:13.785870    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:13.799212 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:47:13.799182    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e82de8_f42b_43de_982c_ff96e03970fd.slice/crio-e10d82cb9b3044d5975a8a90d76772b5872deef83659018d2694e34ba52aeeb3 WatchSource:0}: Error finding container e10d82cb9b3044d5975a8a90d76772b5872deef83659018d2694e34ba52aeeb3: Status 404 returned error can't find the container with id e10d82cb9b3044d5975a8a90d76772b5872deef83659018d2694e34ba52aeeb3
Oct 29 22:47:14.096428 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.096390    4839 generic.go:356] "Generic (PLEG): container finished" podID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerID="94c0fbcd5a831c859cc43159394cf07452a61134cfdc568c8c9207fd366863b6" exitCode=0
Oct 29 22:47:14.096841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.096476    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq" event={"ID":"5cdbf12a-610c-4c00-b552-b80bafafa93b","Type":"ContainerDied","Data":"94c0fbcd5a831c859cc43159394cf07452a61134cfdc568c8c9207fd366863b6"}
Oct 29 22:47:14.096841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.096536    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq" event={"ID":"5cdbf12a-610c-4c00-b552-b80bafafa93b","Type":"ContainerStarted","Data":"895e9fac68cfe6f2173f3930c92791f15c7def28bb6575607235e447464c8b60"}
Oct 29 22:47:14.098081 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.098051    4839 generic.go:356] "Generic (PLEG): container finished" podID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerID="dbd8986544145fbfa022f37ebd47f7190a4cebc0d1190c24fac25d34a9c7373c" exitCode=0
Oct 29 22:47:14.098196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.098179    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7" event={"ID":"340767b7-0ee1-4dab-96c0-e31c16a4cc91","Type":"ContainerDied","Data":"dbd8986544145fbfa022f37ebd47f7190a4cebc0d1190c24fac25d34a9c7373c"}
Oct 29 22:47:14.098264 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.098223    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7" event={"ID":"340767b7-0ee1-4dab-96c0-e31c16a4cc91","Type":"ContainerStarted","Data":"0864e0faa79fd880a71b15acbcea0341e276caf6f33e5259963d7827fa75ae03"}
Oct 29 22:47:14.099683 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.099666    4839 generic.go:356] "Generic (PLEG): container finished" podID="73e82de8-f42b-43de-982c-ff96e03970fd" containerID="50fe1382cea8024dc2eb91b06b96b62db03cb75db3e242072d06e4f40e564698" exitCode=0
Oct 29 22:47:14.099746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.099729    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587" event={"ID":"73e82de8-f42b-43de-982c-ff96e03970fd","Type":"ContainerDied","Data":"50fe1382cea8024dc2eb91b06b96b62db03cb75db3e242072d06e4f40e564698"}
Oct 29 22:47:14.099798 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.099766    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587" event={"ID":"73e82de8-f42b-43de-982c-ff96e03970fd","Type":"ContainerStarted","Data":"e10d82cb9b3044d5975a8a90d76772b5872deef83659018d2694e34ba52aeeb3"}
Oct 29 22:47:14.101442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.101391    4839 generic.go:356] "Generic (PLEG): container finished" podID="dd110d23-f5ae-4543-b782-7420a434db46" containerID="606fba03dd45b2e031b74a461949ef356d6ed2f42dae86268918340799cba03b" exitCode=0
Oct 29 22:47:14.101442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.101412    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s" event={"ID":"dd110d23-f5ae-4543-b782-7420a434db46","Type":"ContainerDied","Data":"606fba03dd45b2e031b74a461949ef356d6ed2f42dae86268918340799cba03b"}
Oct 29 22:47:14.101600 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:14.101448    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s" event={"ID":"dd110d23-f5ae-4543-b782-7420a434db46","Type":"ContainerStarted","Data":"87b7a0f2dabd1d4a003df77a76b4b66b1958f772f36c40e72bbaa1529d6e83ed"}
Oct 29 22:47:15.107439 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.107409    4839 generic.go:356] "Generic (PLEG): container finished" podID="73e82de8-f42b-43de-982c-ff96e03970fd" containerID="7df8a7a2026fc89c27f81f8ea87b8d492f07cf667bc716094b48b2ac6bd63ac6" exitCode=0
Oct 29 22:47:15.107884 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.107500    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587" event={"ID":"73e82de8-f42b-43de-982c-ff96e03970fd","Type":"ContainerDied","Data":"7df8a7a2026fc89c27f81f8ea87b8d492f07cf667bc716094b48b2ac6bd63ac6"}
Oct 29 22:47:15.109431 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.109409    4839 generic.go:356] "Generic (PLEG): container finished" podID="dd110d23-f5ae-4543-b782-7420a434db46" containerID="e3f11f0b635bee92da5155e3ea64f7ad097b86a9c00c1a0b7f9818d61f884db7" exitCode=0
Oct 29 22:47:15.109540 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.109447    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s" event={"ID":"dd110d23-f5ae-4543-b782-7420a434db46","Type":"ContainerDied","Data":"e3f11f0b635bee92da5155e3ea64f7ad097b86a9c00c1a0b7f9818d61f884db7"}
Oct 29 22:47:15.111343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.111326    4839 generic.go:356] "Generic (PLEG): container finished" podID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerID="ddc9921211558e3021e5bed83d0c621a0af35c7466788f5f22ac24907b599411" exitCode=0
Oct 29 22:47:15.111421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.111397    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq" event={"ID":"5cdbf12a-610c-4c00-b552-b80bafafa93b","Type":"ContainerDied","Data":"ddc9921211558e3021e5bed83d0c621a0af35c7466788f5f22ac24907b599411"}
Oct 29 22:47:15.113300 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.113278    4839 generic.go:356] "Generic (PLEG): container finished" podID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerID="ba42671cd8e76bf1a94c0b495fc8dd9b33106e89483ce4334e8d9829ca07fb54" exitCode=0
Oct 29 22:47:15.113412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:15.113314    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7" event={"ID":"340767b7-0ee1-4dab-96c0-e31c16a4cc91","Type":"ContainerDied","Data":"ba42671cd8e76bf1a94c0b495fc8dd9b33106e89483ce4334e8d9829ca07fb54"}
Oct 29 22:47:16.119456 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.119419    4839 generic.go:356] "Generic (PLEG): container finished" podID="73e82de8-f42b-43de-982c-ff96e03970fd" containerID="b0ec707da8f38841e40f723e0d32bd37f14f3638e949b60c5aa990a712f94eb6" exitCode=0
Oct 29 22:47:16.119821 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.119496    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587" event={"ID":"73e82de8-f42b-43de-982c-ff96e03970fd","Type":"ContainerDied","Data":"b0ec707da8f38841e40f723e0d32bd37f14f3638e949b60c5aa990a712f94eb6"}
Oct 29 22:47:16.121456 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.121432    4839 generic.go:356] "Generic (PLEG): container finished" podID="dd110d23-f5ae-4543-b782-7420a434db46" containerID="b9e4d2574d3bad0fa0b79b769ba2737326fbf585897c7d3841a443e45e0dd9fd" exitCode=0
Oct 29 22:47:16.121589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.121510    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s" event={"ID":"dd110d23-f5ae-4543-b782-7420a434db46","Type":"ContainerDied","Data":"b9e4d2574d3bad0fa0b79b769ba2737326fbf585897c7d3841a443e45e0dd9fd"}
Oct 29 22:47:16.123332 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.123309    4839 generic.go:356] "Generic (PLEG): container finished" podID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerID="70c066676b929cb4d2fc560893a2889a9024641e763a447a7a85378235b76b56" exitCode=0
Oct 29 22:47:16.123422 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.123385    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq" event={"ID":"5cdbf12a-610c-4c00-b552-b80bafafa93b","Type":"ContainerDied","Data":"70c066676b929cb4d2fc560893a2889a9024641e763a447a7a85378235b76b56"}
Oct 29 22:47:16.125411 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.125393    4839 generic.go:356] "Generic (PLEG): container finished" podID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerID="25355d85447707cd9f91065b09359bd9e1d4bd0de27ad1f60ad67e6d5c6ad7e5" exitCode=0
Oct 29 22:47:16.125496 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:16.125432    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7" event={"ID":"340767b7-0ee1-4dab-96c0-e31c16a4cc91","Type":"ContainerDied","Data":"25355d85447707cd9f91065b09359bd9e1d4bd0de27ad1f60ad67e6d5c6ad7e5"}
Oct 29 22:47:17.274853 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.274829    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:17.295344 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.295316    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:17.298332 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.298308    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:17.321186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.321140    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:17.324692 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324669    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-bundle\") pod \"5cdbf12a-610c-4c00-b552-b80bafafa93b\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") "
Oct 29 22:47:17.324822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324702    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-util\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-util\") pod \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") "
Oct 29 22:47:17.324822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324725    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd110d23-f5ae-4543-b782-7420a434db46-kube-api-access-vcw4v\" (UniqueName: \"kubernetes.io/projected/dd110d23-f5ae-4543-b782-7420a434db46-kube-api-access-vcw4v\") pod \"dd110d23-f5ae-4543-b782-7420a434db46\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") "
Oct 29 22:47:17.324822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324743    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-bundle\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-bundle\") pod \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") "
Oct 29 22:47:17.324822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324771    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-util\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-util\") pod \"73e82de8-f42b-43de-982c-ff96e03970fd\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") "
Oct 29 22:47:17.324822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324798    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-util\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-util\") pod \"dd110d23-f5ae-4543-b782-7420a434db46\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") "
Oct 29 22:47:17.324822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324815    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340767b7-0ee1-4dab-96c0-e31c16a4cc91-kube-api-access-xpgqn\" (UniqueName: \"kubernetes.io/projected/340767b7-0ee1-4dab-96c0-e31c16a4cc91-kube-api-access-xpgqn\") pod \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\" (UID: \"340767b7-0ee1-4dab-96c0-e31c16a4cc91\") "
Oct 29 22:47:17.325130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324846    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-util\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-util\") pod \"5cdbf12a-610c-4c00-b552-b80bafafa93b\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") "
Oct 29 22:47:17.325130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324880    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cdbf12a-610c-4c00-b552-b80bafafa93b-kube-api-access-4qv8w\" (UniqueName: \"kubernetes.io/projected/5cdbf12a-610c-4c00-b552-b80bafafa93b-kube-api-access-4qv8w\") pod \"5cdbf12a-610c-4c00-b552-b80bafafa93b\" (UID: \"5cdbf12a-610c-4c00-b552-b80bafafa93b\") "
Oct 29 22:47:17.325130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.324940    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-bundle\") pod \"dd110d23-f5ae-4543-b782-7420a434db46\" (UID: \"dd110d23-f5ae-4543-b782-7420a434db46\") "
Oct 29 22:47:17.325577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.325234    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-bundle" pod "5cdbf12a-610c-4c00-b552-b80bafafa93b" (UID: "5cdbf12a-610c-4c00-b552-b80bafafa93b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.325697 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.325676    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-bundle" pod "dd110d23-f5ae-4543-b782-7420a434db46" (UID: "dd110d23-f5ae-4543-b782-7420a434db46"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.327492 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.327465    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd110d23-f5ae-4543-b782-7420a434db46-kube-api-access-vcw4v" pod "dd110d23-f5ae-4543-b782-7420a434db46" (UID: "dd110d23-f5ae-4543-b782-7420a434db46"). InnerVolumeSpecName "kube-api-access-vcw4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:17.327587 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.327557    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340767b7-0ee1-4dab-96c0-e31c16a4cc91-kube-api-access-xpgqn" pod "340767b7-0ee1-4dab-96c0-e31c16a4cc91" (UID: "340767b7-0ee1-4dab-96c0-e31c16a4cc91"). InnerVolumeSpecName "kube-api-access-xpgqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:17.327713 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.327680    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-bundle" pod "340767b7-0ee1-4dab-96c0-e31c16a4cc91" (UID: "340767b7-0ee1-4dab-96c0-e31c16a4cc91"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.329258 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.329233    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cdbf12a-610c-4c00-b552-b80bafafa93b-kube-api-access-4qv8w" pod "5cdbf12a-610c-4c00-b552-b80bafafa93b" (UID: "5cdbf12a-610c-4c00-b552-b80bafafa93b"). InnerVolumeSpecName "kube-api-access-4qv8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:17.330703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.330677    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-util" pod "73e82de8-f42b-43de-982c-ff96e03970fd" (UID: "73e82de8-f42b-43de-982c-ff96e03970fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.331616 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.331593    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-util" pod "340767b7-0ee1-4dab-96c0-e31c16a4cc91" (UID: "340767b7-0ee1-4dab-96c0-e31c16a4cc91"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.334863 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.334830    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-util" pod "5cdbf12a-610c-4c00-b552-b80bafafa93b" (UID: "5cdbf12a-610c-4c00-b552-b80bafafa93b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.335223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.335201    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-util" pod "dd110d23-f5ae-4543-b782-7420a434db46" (UID: "dd110d23-f5ae-4543-b782-7420a434db46"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.425473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425390    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e82de8-f42b-43de-982c-ff96e03970fd-kube-api-access-fjvxh\" (UniqueName: \"kubernetes.io/projected/73e82de8-f42b-43de-982c-ff96e03970fd-kube-api-access-fjvxh\") pod \"73e82de8-f42b-43de-982c-ff96e03970fd\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") "
Oct 29 22:47:17.425473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425435    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-bundle\") pod \"73e82de8-f42b-43de-982c-ff96e03970fd\" (UID: \"73e82de8-f42b-43de-982c-ff96e03970fd\") "
Oct 29 22:47:17.425694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425611    4839 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-bundle\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425629    4839 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-bundle\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425642    4839 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-util\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425655    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcw4v\" (UniqueName: \"kubernetes.io/projected/dd110d23-f5ae-4543-b782-7420a434db46-kube-api-access-vcw4v\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425670    4839 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/340767b7-0ee1-4dab-96c0-e31c16a4cc91-bundle\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425683    4839 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-util\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425694    4839 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd110d23-f5ae-4543-b782-7420a434db46-util\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425706    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpgqn\" (UniqueName: \"kubernetes.io/projected/340767b7-0ee1-4dab-96c0-e31c16a4cc91-kube-api-access-xpgqn\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425718    4839 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5cdbf12a-610c-4c00-b552-b80bafafa93b-util\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.425920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.425732    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qv8w\" (UniqueName: \"kubernetes.io/projected/5cdbf12a-610c-4c00-b552-b80bafafa93b-kube-api-access-4qv8w\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.426050 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.426028    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-bundle" pod "73e82de8-f42b-43de-982c-ff96e03970fd" (UID: "73e82de8-f42b-43de-982c-ff96e03970fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:47:17.427495 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.427475    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e82de8-f42b-43de-982c-ff96e03970fd-kube-api-access-fjvxh" pod "73e82de8-f42b-43de-982c-ff96e03970fd" (UID: "73e82de8-f42b-43de-982c-ff96e03970fd"). InnerVolumeSpecName "kube-api-access-fjvxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:47:17.526645 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.526607    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjvxh\" (UniqueName: \"kubernetes.io/projected/73e82de8-f42b-43de-982c-ff96e03970fd-kube-api-access-fjvxh\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:17.526645 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:17.526640    4839 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/73e82de8-f42b-43de-982c-ff96e03970fd-bundle\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:47:18.134984 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.134952    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587"
Oct 29 22:47:18.135177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.134947    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acp587" event={"ID":"73e82de8-f42b-43de-982c-ff96e03970fd","Type":"ContainerDied","Data":"e10d82cb9b3044d5975a8a90d76772b5872deef83659018d2694e34ba52aeeb3"}
Oct 29 22:47:18.135177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.135094    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e10d82cb9b3044d5975a8a90d76772b5872deef83659018d2694e34ba52aeeb3"
Oct 29 22:47:18.136946 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.136926    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s"
Oct 29 22:47:18.137072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.136957    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctkz4s" event={"ID":"dd110d23-f5ae-4543-b782-7420a434db46","Type":"ContainerDied","Data":"87b7a0f2dabd1d4a003df77a76b4b66b1958f772f36c40e72bbaa1529d6e83ed"}
Oct 29 22:47:18.137072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.137008    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b7a0f2dabd1d4a003df77a76b4b66b1958f772f36c40e72bbaa1529d6e83ed"
Oct 29 22:47:18.138902 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.138871    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq" event={"ID":"5cdbf12a-610c-4c00-b552-b80bafafa93b","Type":"ContainerDied","Data":"895e9fac68cfe6f2173f3930c92791f15c7def28bb6575607235e447464c8b60"}
Oct 29 22:47:18.139014 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.138919    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="895e9fac68cfe6f2173f3930c92791f15c7def28bb6575607235e447464c8b60"
Oct 29 22:47:18.139014 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.138878    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6jctvq"
Oct 29 22:47:18.140905 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.140879    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7" event={"ID":"340767b7-0ee1-4dab-96c0-e31c16a4cc91","Type":"ContainerDied","Data":"0864e0faa79fd880a71b15acbcea0341e276caf6f33e5259963d7827fa75ae03"}
Oct 29 22:47:18.141013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.140920    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v9kj7"
Oct 29 22:47:18.141013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:18.140922    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0864e0faa79fd880a71b15acbcea0341e276caf6f33e5259963d7827fa75ae03"
Oct 29 22:47:21.072803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:21.072762    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:23.579433 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:23.579395    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:23.922834 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:23.922755    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:25.853242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:25.853208    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:26.055806 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:26.055774    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:26.559006 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:26.558968    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:27.268059 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:27.268027    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:28.446914 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:28.446877    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:28.885423 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:28.885380    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:29.142595 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:29.142518    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:29.287656 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:29.287623    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:29.597486 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:29.597453    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:29.746454 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:29.746410    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:33.236631 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:47:33.236595    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:47:34.853851 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:34.853807    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:34.956720 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:34.956692    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:38.835273 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:38.835197    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:38.935699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:38.935667    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:39.234334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:39.234256    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:39.338755 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:39.338723    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:40.644116 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:40.644084    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:40.746505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:40.746473    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:42.029985 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:42.029949    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:42.131202 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:42.131171    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:48.426535 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:48.426504    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:48.546316 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:48.546287    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:48.643439 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:48.643402    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:55.145500 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:55.145464    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:55.385076 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:55.385036    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:58.864660 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:58.864622    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:58.977672 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:58.977636    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:47:59.083493 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:47:59.083461    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:02.845050 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:02.845013    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:02.980301 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:02.980266    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:03.236499 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:48:03.236423    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:48:09.258308 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:09.258267    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:09.368550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:09.368518    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:09.846928 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:09.846892    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:09.959319 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:09.959286    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:10.448398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:10.448366    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:10.555675 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:10.555641    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:13.914847 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:13.914810    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:22.053297 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:22.053257    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:22.218167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:22.218123    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:33.236582 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:48:33.236546    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:48:36.733440 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:36.733404    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:36.841866 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:36.841836    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:36.945265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:36.945235    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:37.047077 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:37.047044    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:37.681065 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:37.681031    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:37.781210 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:37.781177    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:54.323550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:54.323518    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:54.423578 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:54.423545    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:48:55.098966 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:48:55.098927    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:03.236402 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:49:03.236365    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:49:07.580054 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:07.579974    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:07.704197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:07.704142    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:14.023657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:14.023619    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:16.556122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:16.556081    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:16.667442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:16.667414    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:16.766017 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:16.765977    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:16.866270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:16.866194    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:20.537930 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:20.537897    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:20.682590 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:20.682559    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:24.106451 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:24.106410    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:24.228261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:24.228228    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:24.364626 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:24.364542    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:33.237611 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:49:33.237571    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:49:41.522801 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:41.522762    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:52.744599 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:52.744562    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:52.848768 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:52.848732    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:54.046859 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:54.046804    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:54.145196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:54.145167    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:54.245309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:54.245271    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:49:54.341367 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:49:54.341283    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:03.237105 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:50:03.237060    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:50:11.960331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:50:11.960294    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:12.058262 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:50:12.058226    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:14.146577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:50:14.146542    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:33.236853 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:50:33.236818    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:50:38.449456 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:50:38.449379    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:38.552499 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:50:38.552462    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:56.991069 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:50:56.991028    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:50:59.934272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:50:59.934238    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:02.482472 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482436    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"]
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482806    4839 cpu_manager.go:414] "containerMap: removing container" podUID="73e82de8-f42b-43de-982c-ff96e03970fd" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482822    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e82de8-f42b-43de-982c-ff96e03970fd" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482835    4839 cpu_manager.go:414] "containerMap: removing container" podUID="73e82de8-f42b-43de-982c-ff96e03970fd" containerName="pull"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482842    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e82de8-f42b-43de-982c-ff96e03970fd" containerName="pull"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482855    4839 cpu_manager.go:414] "containerMap: removing container" podUID="73e82de8-f42b-43de-982c-ff96e03970fd" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482862    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e82de8-f42b-43de-982c-ff96e03970fd" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482875    4839 cpu_manager.go:414] "containerMap: removing container" podUID="dd110d23-f5ae-4543-b782-7420a434db46" containerName="pull"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482881    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd110d23-f5ae-4543-b782-7420a434db46" containerName="pull"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482891    4839 cpu_manager.go:414] "containerMap: removing container" podUID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerName="pull"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482898    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerName="pull"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482913    4839 cpu_manager.go:414] "containerMap: removing container" podUID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482920    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482936    4839 cpu_manager.go:414] "containerMap: removing container" podUID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482944    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482961    4839 cpu_manager.go:414] "containerMap: removing container" podUID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482968    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482977    4839 cpu_manager.go:414] "containerMap: removing container" podUID="dd110d23-f5ae-4543-b782-7420a434db46" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482984    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd110d23-f5ae-4543-b782-7420a434db46" containerName="extract"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482993    4839 cpu_manager.go:414] "containerMap: removing container" podUID="dd110d23-f5ae-4543-b782-7420a434db46" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.482999    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd110d23-f5ae-4543-b782-7420a434db46" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483007    4839 cpu_manager.go:414] "containerMap: removing container" podUID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483016    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerName="util"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483032    4839 cpu_manager.go:414] "containerMap: removing container" podUID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerName="pull"
Oct 29 22:51:02.483051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483042    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerName="pull"
Oct 29 22:51:02.484382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483100    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e82de8-f42b-43de-982c-ff96e03970fd" containerName="extract"
Oct 29 22:51:02.484382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483113    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd110d23-f5ae-4543-b782-7420a434db46" containerName="extract"
Oct 29 22:51:02.484382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483125    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cdbf12a-610c-4c00-b552-b80bafafa93b" containerName="extract"
Oct 29 22:51:02.484382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.483135    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="340767b7-0ee1-4dab-96c0-e31c16a4cc91" containerName="extract"
Oct 29 22:51:02.486165 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.486123    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.525966 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.525940    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"]
Oct 29 22:51:02.538245 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.538217    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 29 22:51:02.538245 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.538230    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 29 22:51:02.538245 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.538243    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pzhxp\""
Oct 29 22:51:02.538609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.538593    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 29 22:51:02.558529 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.558502    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:02.627739 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.627702    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.627907 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.627750    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ksr\" (UniqueName: \"kubernetes.io/projected/21d37309-aa51-44b7-8fdd-e87e114aaaad-kube-api-access-j8ksr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.627907 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.627858    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.627907 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.627878    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.628014 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.627947    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.628014 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.627983    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d37309-aa51-44b7-8fdd-e87e114aaaad-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.728834 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.728798    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.728834 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.728835    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.729070 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.728860    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.729070 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.728897    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d37309-aa51-44b7-8fdd-e87e114aaaad-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.729070 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.728932    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.729070 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.728980    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ksr\" (UniqueName: \"kubernetes.io/projected/21d37309-aa51-44b7-8fdd-e87e114aaaad-kube-api-access-j8ksr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.729315 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.729246    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.729384 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.729350    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.729428 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.729410    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.731140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.731112    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21d37309-aa51-44b7-8fdd-e87e114aaaad-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.731454 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.731433    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21d37309-aa51-44b7-8fdd-e87e114aaaad-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.750178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.750106    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ksr\" (UniqueName: \"kubernetes.io/projected/21d37309-aa51-44b7-8fdd-e87e114aaaad-kube-api-access-j8ksr\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-t7hn5\" (UID: \"21d37309-aa51-44b7-8fdd-e87e114aaaad\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:02.799069 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:02.799032    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:03.089592 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:03.089569    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"]
Oct 29 22:51:03.090486 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:03.090456    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d37309_aa51_44b7_8fdd_e87e114aaaad.slice/crio-9a1c278680c36bf2f6471abfee9deae7de951fa894f7a44a7557c18168d381d5 WatchSource:0}: Error finding container 9a1c278680c36bf2f6471abfee9deae7de951fa894f7a44a7557c18168d381d5: Status 404 returned error can't find the container with id 9a1c278680c36bf2f6471abfee9deae7de951fa894f7a44a7557c18168d381d5
Oct 29 22:51:03.091816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:03.091794    4839 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:51:03.238910 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:51:03.238871    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:51:03.964975 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:03.964938    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:04.060668 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:04.060632    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5" event={"ID":"21d37309-aa51-44b7-8fdd-e87e114aaaad","Type":"ContainerStarted","Data":"9a1c278680c36bf2f6471abfee9deae7de951fa894f7a44a7557c18168d381d5"}
Oct 29 22:51:05.066606 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:05.066567    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5" event={"ID":"21d37309-aa51-44b7-8fdd-e87e114aaaad","Type":"ContainerStarted","Data":"9757c00f77320d32cdb39530c1bce32e42a7c845d742096a07a74298b53365d5"}
Oct 29 22:51:05.112119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:05.112068    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5" podStartSLOduration=1.88577444 podStartE2EDuration="3.11205517s" totalImagesPullingTime="1.22628073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:05.111722138 +0000 UTC m=+522.214798410" watchObservedRunningTime="2026-10-29 22:51:05.11205517 +0000 UTC m=+522.215131440"
Oct 29 22:51:05.799566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:05.799529    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:07.212365 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.212325    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"]
Oct 29 22:51:07.217068 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.217043    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.235612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.235579    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"]
Oct 29 22:51:07.248138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.248115    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 29 22:51:07.275160 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.275114    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:07.384347 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.384306    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.384515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.384356    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.384515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.384443    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.384515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.384479    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.384632 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.384537    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.384632 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.384576    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrdp\" (UniqueName: \"kubernetes.io/projected/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-kube-api-access-kfrdp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.485556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.485472    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrdp\" (UniqueName: \"kubernetes.io/projected/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-kube-api-access-kfrdp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.485556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.485534    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.485792 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.485560    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.485792 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.485578    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.485792 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.485595    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.485792 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.485613    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.486004 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.485976    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.486073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.486053    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.486141 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.486089    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.487857 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.487825    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.488197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.488177    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.507939 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.507911    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrdp\" (UniqueName: \"kubernetes.io/projected/6b1ce2aa-79c5-4963-84b2-993a8a24dfcb-kube-api-access-kfrdp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h\" (UID: \"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.532520 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.532491    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:07.830087 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:07.830048    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1ce2aa_79c5_4963_84b2_993a8a24dfcb.slice/crio-e833074821019dd9c578aa057679a7f5ee86a0134ad1379e497ead965aa9ac8e WatchSource:0}: Error finding container e833074821019dd9c578aa057679a7f5ee86a0134ad1379e497ead965aa9ac8e: Status 404 returned error can't find the container with id e833074821019dd9c578aa057679a7f5ee86a0134ad1379e497ead965aa9ac8e
Oct 29 22:51:07.830638 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:07.830588    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"]
Oct 29 22:51:08.081554 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:08.081456    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h" event={"ID":"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb","Type":"ContainerStarted","Data":"b926eaa0d018c1b34257ee79a4faa3b955fb4972c308d8478a7798488ae0cac8"}
Oct 29 22:51:08.081554 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:08.081526    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h" event={"ID":"6b1ce2aa-79c5-4963-84b2-993a8a24dfcb","Type":"ContainerStarted","Data":"e833074821019dd9c578aa057679a7f5ee86a0134ad1379e497ead965aa9ac8e"}
Oct 29 22:51:08.126029 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:08.125965    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h" podStartSLOduration=0.95099942 podStartE2EDuration="1.125947107s" totalImagesPullingTime="174.947687ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:08.123953511 +0000 UTC m=+525.227029794" watchObservedRunningTime="2026-10-29 22:51:08.125947107 +0000 UTC m=+525.229023380"
Oct 29 22:51:08.533393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:08.533365    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:08.865097 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:08.865013    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:11.788942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:11.788908    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:12.799906 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:12.799868    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:12.812363 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:12.812336    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:13.114122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:13.114044    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-t7hn5"
Oct 29 22:51:14.275940 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:14.275886    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:17.533130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:17.533096    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:17.545648 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:17.545617    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:18.134820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:18.134791    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h"
Oct 29 22:51:22.776789 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:22.776754    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:29.695118 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.695050    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:29.704592 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.704565    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"]
Oct 29 22:51:29.710699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.710678    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:29.727477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.727452    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"]
Oct 29 22:51:29.739779 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.739753    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 29 22:51:29.740123 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.739782    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 29 22:51:29.740123 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.739828    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-g9sl6\""
Oct 29 22:51:29.763823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.763792    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef1b9f4-455b-424f-834f-62e3b0112246-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-4wcv6\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:29.763959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.763848    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzdw\" (UniqueName: \"kubernetes.io/projected/5ef1b9f4-455b-424f-834f-62e3b0112246-kube-api-access-5pzdw\") pod \"ai-gateway-controller-7d6c696b49-4wcv6\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:29.830484 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.830448    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:29.864498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.864466    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef1b9f4-455b-424f-834f-62e3b0112246-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-4wcv6\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:29.864638 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.864522    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzdw\" (UniqueName: \"kubernetes.io/projected/5ef1b9f4-455b-424f-834f-62e3b0112246-kube-api-access-5pzdw\") pod \"ai-gateway-controller-7d6c696b49-4wcv6\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:29.866952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.866929    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef1b9f4-455b-424f-834f-62e3b0112246-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-4wcv6\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:29.889572 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:29.889546    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzdw\" (UniqueName: \"kubernetes.io/projected/5ef1b9f4-455b-424f-834f-62e3b0112246-kube-api-access-5pzdw\") pod \"ai-gateway-controller-7d6c696b49-4wcv6\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:30.023256 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:30.023220    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:30.316948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:30.316920    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"]
Oct 29 22:51:30.318111 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:30.318084    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef1b9f4_455b_424f_834f_62e3b0112246.slice/crio-f59ce0fbdf5ec7c623a10a4ec61d572fb7dc063e90fd861e32824bc55c7bac10 WatchSource:0}: Error finding container f59ce0fbdf5ec7c623a10a4ec61d572fb7dc063e90fd861e32824bc55c7bac10: Status 404 returned error can't find the container with id f59ce0fbdf5ec7c623a10a4ec61d572fb7dc063e90fd861e32824bc55c7bac10
Oct 29 22:51:31.178166 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:31.178118    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6" event={"ID":"5ef1b9f4-455b-424f-834f-62e3b0112246","Type":"ContainerStarted","Data":"f59ce0fbdf5ec7c623a10a4ec61d572fb7dc063e90fd861e32824bc55c7bac10"}
Oct 29 22:51:33.188343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:33.188306    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6" event={"ID":"5ef1b9f4-455b-424f-834f-62e3b0112246","Type":"ContainerStarted","Data":"67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8"}
Oct 29 22:51:33.188784 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:33.188382    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:33.231934 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:33.231881    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6" podStartSLOduration=1.858380862 podStartE2EDuration="4.231866314s" totalImagesPullingTime="2.373485452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:33.231347426 +0000 UTC m=+550.334423698" watchObservedRunningTime="2026-10-29 22:51:33.231866314 +0000 UTC m=+550.334942586"
Oct 29 22:51:33.238276 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:51:33.238245    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:51:34.798719 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:34.798686    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:34.898982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:34.898945    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:34.903829 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:34.903801    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-6bfdff5577-nthkr"]
Oct 29 22:51:34.907090 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:34.907072    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:34.926879 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:34.926850    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bfdff5577-nthkr"]
Oct 29 22:51:34.927350 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:34.927311    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 29 22:51:34.938253 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:34.938224    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2d967\""
Oct 29 22:51:35.009544 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.009515    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bbe52f-800c-4843-a6ab-9ce20cb3a508-plugins-config-volume\") pod \"payload-processing-6bfdff5577-nthkr\" (UID: \"00bbe52f-800c-4843-a6ab-9ce20cb3a508\") " pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:35.009703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.009615    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljqwg\" (UniqueName: \"kubernetes.io/projected/00bbe52f-800c-4843-a6ab-9ce20cb3a508-kube-api-access-ljqwg\") pod \"payload-processing-6bfdff5577-nthkr\" (UID: \"00bbe52f-800c-4843-a6ab-9ce20cb3a508\") " pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:35.110296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.110215    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljqwg\" (UniqueName: \"kubernetes.io/projected/00bbe52f-800c-4843-a6ab-9ce20cb3a508-kube-api-access-ljqwg\") pod \"payload-processing-6bfdff5577-nthkr\" (UID: \"00bbe52f-800c-4843-a6ab-9ce20cb3a508\") " pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:35.110296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.110287    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bbe52f-800c-4843-a6ab-9ce20cb3a508-plugins-config-volume\") pod \"payload-processing-6bfdff5577-nthkr\" (UID: \"00bbe52f-800c-4843-a6ab-9ce20cb3a508\") " pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:35.111012 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.110990    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bbe52f-800c-4843-a6ab-9ce20cb3a508-plugins-config-volume\") pod \"payload-processing-6bfdff5577-nthkr\" (UID: \"00bbe52f-800c-4843-a6ab-9ce20cb3a508\") " pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:35.130987 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.130953    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljqwg\" (UniqueName: \"kubernetes.io/projected/00bbe52f-800c-4843-a6ab-9ce20cb3a508-kube-api-access-ljqwg\") pod \"payload-processing-6bfdff5577-nthkr\" (UID: \"00bbe52f-800c-4843-a6ab-9ce20cb3a508\") " pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:35.220289 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.220255    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:35.515821 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:35.515797    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-6bfdff5577-nthkr"]
Oct 29 22:51:35.516986 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:35.516952    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bbe52f_800c_4843_a6ab_9ce20cb3a508.slice/crio-6765df31489cdb820c17fe86d571a57b21b9304201da496fe51747656fbd5875 WatchSource:0}: Error finding container 6765df31489cdb820c17fe86d571a57b21b9304201da496fe51747656fbd5875: Status 404 returned error can't find the container with id 6765df31489cdb820c17fe86d571a57b21b9304201da496fe51747656fbd5875
Oct 29 22:51:36.204657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:36.204619    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bfdff5577-nthkr" event={"ID":"00bbe52f-800c-4843-a6ab-9ce20cb3a508","Type":"ContainerStarted","Data":"6765df31489cdb820c17fe86d571a57b21b9304201da496fe51747656fbd5875"}
Oct 29 22:51:37.848479 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.848438    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"]
Oct 29 22:51:37.851510 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.851490    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.878515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.878481    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"]
Oct 29 22:51:37.934985 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.934940    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqpc\" (UniqueName: \"kubernetes.io/projected/e42f233c-1cec-4162-abdb-2355ee0fb03a-kube-api-access-5vqpc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935002    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935048    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42f233c-1cec-4162-abdb-2355ee0fb03a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935130    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935183    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935260    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935283    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935305    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.935447 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.935329    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:37.955241 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:37.955210    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z74sk\""
Oct 29 22:51:38.036744 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036706    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.036933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036746    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.036933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036779    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.036933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036812    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.036933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036878    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqpc\" (UniqueName: \"kubernetes.io/projected/e42f233c-1cec-4162-abdb-2355ee0fb03a-kube-api-access-5vqpc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.036933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036911    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037322 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036933    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42f233c-1cec-4162-abdb-2355ee0fb03a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037322 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.036961    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037322 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.037070    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037488 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.037351    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037488 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.037461    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037592 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.037564    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.037599    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.037947 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.037919    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e42f233c-1cec-4162-abdb-2355ee0fb03a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.039273 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.039250    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.039475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.039455    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.059198 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.059171    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqpc\" (UniqueName: \"kubernetes.io/projected/e42f233c-1cec-4162-abdb-2355ee0fb03a-kube-api-access-5vqpc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.060990 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.060973    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e42f233c-1cec-4162-abdb-2355ee0fb03a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hqqmd\" (UID: \"e42f233c-1cec-4162-abdb-2355ee0fb03a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.163744 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.163659    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:38.220019 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.219984    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-6bfdff5577-nthkr" event={"ID":"00bbe52f-800c-4843-a6ab-9ce20cb3a508","Type":"ContainerStarted","Data":"ba8cd926a1c208c2e8b4935bfd7fb764aa1c0227583f744da19fe10f12d29522"}
Oct 29 22:51:38.220679 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.220662    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:38.265181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.265107    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-6bfdff5577-nthkr" podStartSLOduration=2.346611916 podStartE2EDuration="4.265093755s" totalImagesPullingTime="1.918481839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:38.26383422 +0000 UTC m=+555.366910492" watchObservedRunningTime="2026-10-29 22:51:38.265093755 +0000 UTC m=+555.368170178"
Oct 29 22:51:38.472620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:38.472596    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"]
Oct 29 22:51:38.473794 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:38.473760    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42f233c_1cec_4162_abdb_2355ee0fb03a.slice/crio-69fdd52abc314beac1b936fed068665198d67fc3d5c4dfdf653ebd0dd96b7ab8 WatchSource:0}: Error finding container 69fdd52abc314beac1b936fed068665198d67fc3d5c4dfdf653ebd0dd96b7ab8: Status 404 returned error can't find the container with id 69fdd52abc314beac1b936fed068665198d67fc3d5c4dfdf653ebd0dd96b7ab8
Oct 29 22:51:39.226350 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:39.226311    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" event={"ID":"e42f233c-1cec-4162-abdb-2355ee0fb03a","Type":"ContainerStarted","Data":"69fdd52abc314beac1b936fed068665198d67fc3d5c4dfdf653ebd0dd96b7ab8"}
Oct 29 22:51:40.855233 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:40.855204    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:40.855499 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:40.855272    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:40.855499 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:40.855299    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:41.236709 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:41.236599    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" event={"ID":"e42f233c-1cec-4162-abdb-2355ee0fb03a","Type":"ContainerStarted","Data":"2102e7f0cafecf40d96b0b3546cee45eae27405691d84b7c0705317fcdfbc205"}
Oct 29 22:51:42.164333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:42.164293    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:42.165718 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:42.165693    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:51:42.165791 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:42.165743    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 29 22:51:43.164676 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:43.164640    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:51:43.165083 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:43.164699    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 29 22:51:44.164777 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.164743    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:51:44.165199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.164798    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 29 22:51:44.196242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.196210    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:44.246458 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.246407    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podStartSLOduration=4.868183321 podStartE2EDuration="7.246390991s" totalImagesPullingTime="2.37820767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:41.310446295 +0000 UTC m=+558.413522566" watchObservedRunningTime="2026-10-29 22:51:44.246390991 +0000 UTC m=+561.349467262"
Oct 29 22:51:44.495694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.495516    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-d74798bcb-jpzf4"]
Oct 29 22:51:44.513974 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.498978    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:44.520430 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.520403    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d74798bcb-jpzf4"]
Oct 29 22:51:44.616288 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.616247    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5wvh\" (UniqueName: \"kubernetes.io/projected/0be7d699-ee3d-4d60-a663-c7a8730407ec-kube-api-access-n5wvh\") pod \"ai-gateway-controller-d74798bcb-jpzf4\" (UID: \"0be7d699-ee3d-4d60-a663-c7a8730407ec\") " pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:44.616474 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.616305    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0be7d699-ee3d-4d60-a663-c7a8730407ec-metrics-certs\") pod \"ai-gateway-controller-d74798bcb-jpzf4\" (UID: \"0be7d699-ee3d-4d60-a663-c7a8730407ec\") " pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:44.717039 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.717004    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0be7d699-ee3d-4d60-a663-c7a8730407ec-metrics-certs\") pod \"ai-gateway-controller-d74798bcb-jpzf4\" (UID: \"0be7d699-ee3d-4d60-a663-c7a8730407ec\") " pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:44.717266 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.717081    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5wvh\" (UniqueName: \"kubernetes.io/projected/0be7d699-ee3d-4d60-a663-c7a8730407ec-kube-api-access-n5wvh\") pod \"ai-gateway-controller-d74798bcb-jpzf4\" (UID: \"0be7d699-ee3d-4d60-a663-c7a8730407ec\") " pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:44.719582 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.719557    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0be7d699-ee3d-4d60-a663-c7a8730407ec-metrics-certs\") pod \"ai-gateway-controller-d74798bcb-jpzf4\" (UID: \"0be7d699-ee3d-4d60-a663-c7a8730407ec\") " pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:44.738810 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.738784    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5wvh\" (UniqueName: \"kubernetes.io/projected/0be7d699-ee3d-4d60-a663-c7a8730407ec-kube-api-access-n5wvh\") pod \"ai-gateway-controller-d74798bcb-jpzf4\" (UID: \"0be7d699-ee3d-4d60-a663-c7a8730407ec\") " pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:44.811355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:44.811308    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:45.130130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:45.130101    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-d74798bcb-jpzf4"]
Oct 29 22:51:45.131170 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:45.131117    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be7d699_ee3d_4d60_a663_c7a8730407ec.slice/crio-ba153eea02c1859a8f7f9fee2488a9305861e6ca5076d94f46c96560d6ac8f8e WatchSource:0}: Error finding container ba153eea02c1859a8f7f9fee2488a9305861e6ca5076d94f46c96560d6ac8f8e: Status 404 returned error can't find the container with id ba153eea02c1859a8f7f9fee2488a9305861e6ca5076d94f46c96560d6ac8f8e
Oct 29 22:51:45.164477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:45.164441    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:51:45.164620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:45.164493    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 29 22:51:45.254534 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:45.254493    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4" event={"ID":"0be7d699-ee3d-4d60-a663-c7a8730407ec","Type":"ContainerStarted","Data":"ba153eea02c1859a8f7f9fee2488a9305861e6ca5076d94f46c96560d6ac8f8e"}
Oct 29 22:51:46.164757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:46.164719    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:51:46.164942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:46.164774    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 29 22:51:46.260476 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:46.260441    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4" event={"ID":"0be7d699-ee3d-4d60-a663-c7a8730407ec","Type":"ContainerStarted","Data":"32cbd35df76c5392a99cf1f3d299b444f7aa55d01af25243ab787f2f6012fb1c"}
Oct 29 22:51:46.260857 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:46.260563    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:46.309302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:46.309247    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4" podStartSLOduration=2.017504906 podStartE2EDuration="2.3092326s" totalImagesPullingTime="291.727694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:46.307838432 +0000 UTC m=+563.410914715" watchObservedRunningTime="2026-10-29 22:51:46.3092326 +0000 UTC m=+563.412308870"
Oct 29 22:51:47.164989 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:47.164956    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:51:47.165179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:47.165008    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 29 22:51:48.164830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:48.164793    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 29 22:51:48.165242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:48.164850    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 29 22:51:49.168126 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:49.168091    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:49.168540 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:49.168411    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:49.169019 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:49.169002    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd"
Oct 29 22:51:50.234923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:50.234897    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-6bfdff5577-nthkr"
Oct 29 22:51:52.896436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.896395    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"]
Oct 29 22:51:52.899990 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.899965    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.921434 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.921404    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"]
Oct 29 22:51:52.934138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.934107    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"]
Oct 29 22:51:52.937409 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.937392    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:52.960600 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.960572    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"]
Oct 29 22:51:52.986217 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986185    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986224    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986314    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986449 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986374    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f4fedc-6afb-42e4-87de-8b77868b9125-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986449 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986413    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986456    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986485    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986575 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986514    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:52.986575 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:52.986552    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zlp\" (UniqueName: \"kubernetes.io/projected/80f4fedc-6afb-42e4-87de-8b77868b9125-kube-api-access-l8zlp\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.087747 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.087710    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.087747 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.087757    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.087998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.087937    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zlp\" (UniqueName: \"kubernetes.io/projected/80f4fedc-6afb-42e4-87de-8b77868b9125-kube-api-access-l8zlp\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.087998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.087993    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088024    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088090    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088248 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088118    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4957a03-1161-47a8-a4c1-69c75f901597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088248 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088165    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088248 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088217    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088287    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088343    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8tt\" (UniqueName: \"kubernetes.io/projected/f4957a03-1161-47a8-a4c1-69c75f901597-kube-api-access-zn8tt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088394    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088428    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f4fedc-6afb-42e4-87de-8b77868b9125-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088456    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4957a03-1161-47a8-a4c1-69c75f901597-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088491    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088525    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088698 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088558    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088698 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088574    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4957a03-1161-47a8-a4c1-69c75f901597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.088698 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088662    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088701    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.088814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.088795    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.089035 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.089009    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.089085    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f4fedc-6afb-42e4-87de-8b77868b9125-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.090716 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.090684    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.090814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.090723    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.109072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.109042    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f4fedc-6afb-42e4-87de-8b77868b9125-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.109218 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.109140    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zlp\" (UniqueName: \"kubernetes.io/projected/80f4fedc-6afb-42e4-87de-8b77868b9125-kube-api-access-l8zlp\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rntkv\" (UID: \"80f4fedc-6afb-42e4-87de-8b77868b9125\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.189608 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189516    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189608 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189561    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189608 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189581    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4957a03-1161-47a8-a4c1-69c75f901597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189608 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189609    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189941 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189629    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8tt\" (UniqueName: \"kubernetes.io/projected/f4957a03-1161-47a8-a4c1-69c75f901597-kube-api-access-zn8tt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189941 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189658    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4957a03-1161-47a8-a4c1-69c75f901597-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189941 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189704    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189941 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189730    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.189941 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.189752    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4957a03-1161-47a8-a4c1-69c75f901597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.190224 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.190066    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.190224 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.190083    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.190314 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.190294    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.190373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.190309    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.190659 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.190637    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4957a03-1161-47a8-a4c1-69c75f901597-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.192176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.192142    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4957a03-1161-47a8-a4c1-69c75f901597-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.194360 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.194341    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4957a03-1161-47a8-a4c1-69c75f901597-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.211629 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.211570    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4957a03-1161-47a8-a4c1-69c75f901597-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.211761 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.211631    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8tt\" (UniqueName: \"kubernetes.io/projected/f4957a03-1161-47a8-a4c1-69c75f901597-kube-api-access-zn8tt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-84ct6\" (UID: \"f4957a03-1161-47a8-a4c1-69c75f901597\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.216197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.216170    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:53.252645 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.252384    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:53.529982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.529953    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"]
Oct 29 22:51:53.530318 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:53.530287    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f4fedc_6afb_42e4_87de_8b77868b9125.slice/crio-5f05283000202ab232aaef157ede42d0e135d3fc231b649713f5dca3ba874fb0 WatchSource:0}: Error finding container 5f05283000202ab232aaef157ede42d0e135d3fc231b649713f5dca3ba874fb0: Status 404 returned error can't find the container with id 5f05283000202ab232aaef157ede42d0e135d3fc231b649713f5dca3ba874fb0
Oct 29 22:51:53.532336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.532305    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:53.532412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.532371    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:53.532412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.532399    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:53.565376 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.565351    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"]
Oct 29 22:51:53.574287 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:51:53.574258    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4957a03_1161_47a8_a4c1_69c75f901597.slice/crio-22e559b3479896735881e7a02f2be8baea565ef3db574af95e3b74e363cd7908 WatchSource:0}: Error finding container 22e559b3479896735881e7a02f2be8baea565ef3db574af95e3b74e363cd7908: Status 404 returned error can't find the container with id 22e559b3479896735881e7a02f2be8baea565ef3db574af95e3b74e363cd7908
Oct 29 22:51:53.576692 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.576652    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:53.576771 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.576745    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:53.576807 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:53.576795    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:51:54.324583 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:54.324546    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" event={"ID":"80f4fedc-6afb-42e4-87de-8b77868b9125","Type":"ContainerStarted","Data":"4d529b027d3d35a44e59947c7874f1ed5fa8644c04c8b2e26cbe0dafc6ebab23"}
Oct 29 22:51:54.325082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:54.324616    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" event={"ID":"80f4fedc-6afb-42e4-87de-8b77868b9125","Type":"ContainerStarted","Data":"5f05283000202ab232aaef157ede42d0e135d3fc231b649713f5dca3ba874fb0"}
Oct 29 22:51:54.326856 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:54.326835    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" event={"ID":"f4957a03-1161-47a8-a4c1-69c75f901597","Type":"ContainerStarted","Data":"223aea5bae4eedcc565f14600a1558403154cd431e8bec29cc2ba7465527c674"}
Oct 29 22:51:54.327023 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:54.327007    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" event={"ID":"f4957a03-1161-47a8-a4c1-69c75f901597","Type":"ContainerStarted","Data":"22e559b3479896735881e7a02f2be8baea565ef3db574af95e3b74e363cd7908"}
Oct 29 22:51:54.445506 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:54.445459    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podStartSLOduration=2.44544469 podStartE2EDuration="2.44544469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:54.444601568 +0000 UTC m=+571.547677839" watchObservedRunningTime="2026-10-29 22:51:54.44544469 +0000 UTC m=+571.548520961"
Oct 29 22:51:54.446619 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:54.446580    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podStartSLOduration=2.446567697 podStartE2EDuration="2.446567697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:51:54.394312644 +0000 UTC m=+571.497388938" watchObservedRunningTime="2026-10-29 22:51:54.446567697 +0000 UTC m=+571.549643968"
Oct 29 22:51:55.218188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:55.217230    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:51:55.218188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:55.218033    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rntkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:51:55.218188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:55.218090    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podUID="80f4fedc-6afb-42e4-87de-8b77868b9125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 29 22:51:55.253542 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:55.253514    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:51:55.254567 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:55.254541    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-84ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:55.254687 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:55.254585    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podUID="f4957a03-1161-47a8-a4c1-69c75f901597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 29 22:51:56.217522 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:56.217489    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rntkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:51:56.218051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:56.218025    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podUID="80f4fedc-6afb-42e4-87de-8b77868b9125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 29 22:51:56.253512 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:56.253481    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-84ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:56.253669 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:56.253530    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podUID="f4957a03-1161-47a8-a4c1-69c75f901597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 29 22:51:57.217117 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.216617    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rntkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:51:57.217117 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.216680    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podUID="80f4fedc-6afb-42e4-87de-8b77868b9125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 29 22:51:57.253528 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.253494    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-84ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:57.253901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.253544    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podUID="f4957a03-1161-47a8-a4c1-69c75f901597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 29 22:51:57.268412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.268382    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-d74798bcb-jpzf4"
Oct 29 22:51:57.352789 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.352755    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"]
Oct 29 22:51:57.353101 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.353075    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6" podUID="5ef1b9f4-455b-424f-834f-62e3b0112246" containerName="manager" containerID="cri-o://67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8" gracePeriod=30
Oct 29 22:51:57.735182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.735158    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:57.843455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.843420    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ef1b9f4-455b-424f-834f-62e3b0112246-kube-api-access-5pzdw\" (UniqueName: \"kubernetes.io/projected/5ef1b9f4-455b-424f-834f-62e3b0112246-kube-api-access-5pzdw\") pod \"5ef1b9f4-455b-424f-834f-62e3b0112246\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") "
Oct 29 22:51:57.843653 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.843480    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ef1b9f4-455b-424f-834f-62e3b0112246-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef1b9f4-455b-424f-834f-62e3b0112246-metrics-certs\") pod \"5ef1b9f4-455b-424f-834f-62e3b0112246\" (UID: \"5ef1b9f4-455b-424f-834f-62e3b0112246\") "
Oct 29 22:51:57.845770 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.845740    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ef1b9f4-455b-424f-834f-62e3b0112246-metrics-certs" pod "5ef1b9f4-455b-424f-834f-62e3b0112246" (UID: "5ef1b9f4-455b-424f-834f-62e3b0112246"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:51:57.845908 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.845880    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef1b9f4-455b-424f-834f-62e3b0112246-kube-api-access-5pzdw" pod "5ef1b9f4-455b-424f-834f-62e3b0112246" (UID: "5ef1b9f4-455b-424f-834f-62e3b0112246"). InnerVolumeSpecName "kube-api-access-5pzdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:51:57.944671 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.944628    4839 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ef1b9f4-455b-424f-834f-62e3b0112246-metrics-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:51:57.944671 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:57.944665    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pzdw\" (UniqueName: \"kubernetes.io/projected/5ef1b9f4-455b-424f-834f-62e3b0112246-kube-api-access-5pzdw\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:51:58.217993 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.217554    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rntkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:51:58.217993 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.217623    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podUID="80f4fedc-6afb-42e4-87de-8b77868b9125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 29 22:51:58.253042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.253013    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-84ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:58.253215 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.253069    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podUID="f4957a03-1161-47a8-a4c1-69c75f901597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 29 22:51:58.347718 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.347684    4839 generic.go:356] "Generic (PLEG): container finished" podID="5ef1b9f4-455b-424f-834f-62e3b0112246" containerID="67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8" exitCode=0
Oct 29 22:51:58.348221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.347741    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"
Oct 29 22:51:58.348221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.347762    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6" event={"ID":"5ef1b9f4-455b-424f-834f-62e3b0112246","Type":"ContainerDied","Data":"67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8"}
Oct 29 22:51:58.348221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.347800    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-4wcv6" event={"ID":"5ef1b9f4-455b-424f-834f-62e3b0112246","Type":"ContainerDied","Data":"f59ce0fbdf5ec7c623a10a4ec61d572fb7dc063e90fd861e32824bc55c7bac10"}
Oct 29 22:51:58.348221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.347819    4839 scope.go:122] "RemoveContainer" containerID="67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8"
Oct 29 22:51:58.358030 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.358012    4839 scope.go:122] "RemoveContainer" containerID="67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8"
Oct 29 22:51:58.358365 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:51:58.358343    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8\": container with ID starting with 67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8 not found: ID does not exist" containerID="67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8"
Oct 29 22:51:58.358434 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.358373    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8"} err="failed to get container status \"67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8\": rpc error: code = NotFound desc = could not find container \"67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8\": container with ID starting with 67dabb74bfff13645091d8c8abb38c1de2aed880c58c1c8398b7fa8d4fd4f3a8 not found: ID does not exist"
Oct 29 22:51:58.418173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.408131    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"]
Oct 29 22:51:58.421477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:58.421390    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-4wcv6"]
Oct 29 22:51:59.017799 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:59.017635    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:51:59.020848 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:59.020815    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ef1b9f4-455b-424f-834f-62e3b0112246" path="/var/lib/kubelet/pods/5ef1b9f4-455b-424f-834f-62e3b0112246/volumes"
Oct 29 22:51:59.217605 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:59.217564    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rntkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:51:59.217785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:59.217634    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podUID="80f4fedc-6afb-42e4-87de-8b77868b9125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 29 22:51:59.252855 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:59.252820    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-84ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:51:59.253020 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:59.252870    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podUID="f4957a03-1161-47a8-a4c1-69c75f901597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 29 22:51:59.767199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:51:59.767139    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:00.216827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:00.216749    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rntkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 29 22:52:00.216827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:00.216804    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podUID="80f4fedc-6afb-42e4-87de-8b77868b9125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 29 22:52:00.253110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:00.253074    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-84ct6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 29 22:52:00.253268 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:00.253131    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podUID="f4957a03-1161-47a8-a4c1-69c75f901597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 29 22:52:01.129111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:01.129069    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:01.230479 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:01.230441    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:01.995222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:01.995163    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:52:01.995580 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:01.995555    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:52:01.996401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:01.996381    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv"
Oct 29 22:52:02.070687 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:02.070655    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:52:02.071011 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:02.070986    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:52:02.071837 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:02.071801    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6"
Oct 29 22:52:02.360027 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:02.359994    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:02.623917 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:02.623818    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:02.723830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:02.723790    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:03.238581 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:52:03.238548    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:52:14.365400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:14.365304    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:14.464702 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:14.464667    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:14.547859 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:14.547829    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.205138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.205056    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.307017 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.306968    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.662852 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.662812    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"]
Oct 29 22:52:24.663394 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.663376    4839 cpu_manager.go:414] "containerMap: removing container" podUID="5ef1b9f4-455b-424f-834f-62e3b0112246" containerName="manager"
Oct 29 22:52:24.663471 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.663395    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ef1b9f4-455b-424f-834f-62e3b0112246" containerName="manager"
Oct 29 22:52:24.663471 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.663455    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ef1b9f4-455b-424f-834f-62e3b0112246" containerName="manager"
Oct 29 22:52:24.667885 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.667864    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.703381 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.703353    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-20f88f-openshift-default-dockercfg-7q76j\""
Oct 29 22:52:24.706184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.706122    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"]
Oct 29 22:52:24.740691 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.740658    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.748845 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.748812    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.748987 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.748854    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.748987 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.748871    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.748987 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.748964    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75dp\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-kube-api-access-g75dp\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.749121 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.749005    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.749121 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.749026    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.749121 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.749105    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.749267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.749126    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c17881-8e31-4bc4-a55e-6005c5225912-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.749267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.749172    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c17881-8e31-4bc4-a55e-6005c5225912-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850048    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850281 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850084    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850281 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850109    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c17881-8e31-4bc4-a55e-6005c5225912-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850281 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850136    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c17881-8e31-4bc4-a55e-6005c5225912-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850311    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850382    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850410    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850556    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850582    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75dp\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-kube-api-access-g75dp\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850723 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850698    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850810 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850786    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.850879 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.850861    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.851165 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.851019    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.851466 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.851446    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c17881-8e31-4bc4-a55e-6005c5225912-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.853048 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.853006    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.853323 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.853306    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c17881-8e31-4bc4-a55e-6005c5225912-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.875923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.875896    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.876021 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.875921    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75dp\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-kube-api-access-g75dp\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.882574 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.882496    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:24.958179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.958072    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"]
Oct 29 22:52:24.962688 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.962668    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:24.980469 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.980440    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:24.984223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.984200    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"]
Oct 29 22:52:24.987792 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:24.987768    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-81f92d-openshift-default-dockercfg-x8lwz\""
Oct 29 22:52:25.014589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.014553    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.058577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058535    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.058753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058633    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.058753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058660    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.058753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058688    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.058753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058709    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.058753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058736    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.059043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058795    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.059043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058851    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24kw\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-kube-api-access-w24kw\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.059043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.058879    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.119425 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.119389    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.159932 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.159900    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.159943    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24kw\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-kube-api-access-w24kw\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.159961    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160054    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160094    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160110    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160130    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160180    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160209    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160567 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160372    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160567 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160511    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160667 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160620    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160720 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160659    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.160916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.160899    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.162505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.162485    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.162705 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.162689    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.181402 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.181376    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.181523 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.181506    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24kw\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-kube-api-access-w24kw\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.215287 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.215217    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.251948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.251914    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"]
Oct 29 22:52:25.256981 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.256957    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.276318 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.276291    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:25.277576 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.277553    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"]
Oct 29 22:52:25.301774 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.301752    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-a2e7c8-openshift-default-dockercfg-lq5n9\""
Oct 29 22:52:25.309326 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:25.309286    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c17881_8e31_4bc4_a55e_6005c5225912.slice/crio-5554a09aafea1d22886510f939b58e4d4935265f6356285deab9ea34a582bd9b WatchSource:0}: Error finding container 5554a09aafea1d22886510f939b58e4d4935265f6356285deab9ea34a582bd9b: Status 404 returned error can't find the container with id 5554a09aafea1d22886510f939b58e4d4935265f6356285deab9ea34a582bd9b
Oct 29 22:52:25.311855 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.311811    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"]
Oct 29 22:52:25.311948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.311889    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.311948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.311944    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.312019 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.311972    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.335178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.332863    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.362479 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362446    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362488    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362510    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362716 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362649    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362716 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362695    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362795 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362721    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362795 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362753    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vccqv\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-kube-api-access-vccqv\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362903    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.362971 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.362957    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.464399 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464362    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.464572 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464404    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.464572 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464525    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vccqv\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-kube-api-access-vccqv\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.464860 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464769    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.464860 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464841    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464878    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464913    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464939    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.464991    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.465227    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465651 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.465605    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.465880    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.465942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.465894    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.466103 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.466045    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.466772 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.466752    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.468159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.468133    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.474208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.474176    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" event={"ID":"11c17881-8e31-4bc4-a55e-6005c5225912","Type":"ContainerStarted","Data":"61e15854a4450b37531bc037bf56e680feff016aa0027daa1088529d478863bd"}
Oct 29 22:52:25.474358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.474236    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" event={"ID":"11c17881-8e31-4bc4-a55e-6005c5225912","Type":"ContainerStarted","Data":"5554a09aafea1d22886510f939b58e4d4935265f6356285deab9ea34a582bd9b"}
Oct 29 22:52:25.476043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.476021    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.486448 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.486414    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.486625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.486606    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vccqv\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-kube-api-access-vccqv\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.524474 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.524265    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podStartSLOduration=1.524248431 podStartE2EDuration="1.524248431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:25.522922254 +0000 UTC m=+602.625998541" watchObservedRunningTime="2026-10-29 22:52:25.524248431 +0000 UTC m=+602.627324703"
Oct 29 22:52:25.538852 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.538789    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"]
Oct 29 22:52:25.544047 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.544021    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.569491 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.569454    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"]
Oct 29 22:52:25.569798 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.569776    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-dd3ff3-openshift-default-dockercfg-528h2\""
Oct 29 22:52:25.570426 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.570401    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:25.600695 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.600663    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"]
Oct 29 22:52:25.601926 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.601844    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.602983 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:25.602803    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41b48a5_2064_4e12_87b5_4c8b3fb8694c.slice/crio-99a072f8d260b5fb185b6e4f2599585fa4eb1808fa18707375b72c7aa8f05c11 WatchSource:0}: Error finding container 99a072f8d260b5fb185b6e4f2599585fa4eb1808fa18707375b72c7aa8f05c11: Status 404 returned error can't find the container with id 99a072f8d260b5fb185b6e4f2599585fa4eb1808fa18707375b72c7aa8f05c11
Oct 29 22:52:25.609717 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.609684    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.609819 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.609761    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.609819 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.609807    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679634    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679685    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679729    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679799    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679844    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ad91ce-c59e-4a72-9369-018e882bff72-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679882    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679931    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmmpn\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-kube-api-access-vmmpn\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679960    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.681820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.679991    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ad91ce-c59e-4a72-9369-018e882bff72-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.727055 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.726968    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.781037 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.780997    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781037 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781045    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ad91ce-c59e-4a72-9369-018e882bff72-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781072    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781104    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmmpn\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-kube-api-access-vmmpn\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781121    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781140    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ad91ce-c59e-4a72-9369-018e882bff72-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781192    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781207    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.781328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781227    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.782709 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.781839    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.782709 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.782385    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ad91ce-c59e-4a72-9369-018e882bff72-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.782981 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.782953    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.783917 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.783303    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.783917 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.783553    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.785086 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.785059    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.786051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.786030    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ad91ce-c59e-4a72-9369-018e882bff72-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.803241 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.803217    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.806250 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.806228    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmmpn\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-kube-api-access-vmmpn\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.824468 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.824432    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:25.861109 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.861076    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:25.885641 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:25.885597    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68408cd1_eda7_4df5_80ad_e82f9cf809e9.slice/crio-1db358bb08fcea0f62140463cd8bc0e346b3bd2e9ad961bb63c3b1a1c64963b6 WatchSource:0}: Error finding container 1db358bb08fcea0f62140463cd8bc0e346b3bd2e9ad961bb63c3b1a1c64963b6: Status 404 returned error can't find the container with id 1db358bb08fcea0f62140463cd8bc0e346b3bd2e9ad961bb63c3b1a1c64963b6
Oct 29 22:52:25.885852 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.885827    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"]
Oct 29 22:52:25.888689 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.888651    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.888793 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.888728    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.888793 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.888767    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:25.980873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:25.980843    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:26.176773 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:26.176736    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ad91ce_c59e_4a72_9369_018e882bff72.slice/crio-f1d6b1634bae0e140654718c49d833aff117bcce37f6514d36efae63ff56e7fe WatchSource:0}: Error finding container f1d6b1634bae0e140654718c49d833aff117bcce37f6514d36efae63ff56e7fe: Status 404 returned error can't find the container with id f1d6b1634bae0e140654718c49d833aff117bcce37f6514d36efae63ff56e7fe
Oct 29 22:52:26.177812 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.177738    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"]
Oct 29 22:52:26.179677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.179640    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:26.179751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.179723    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:26.179795 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.179765    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:26.481375 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.481330    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" event={"ID":"52ad91ce-c59e-4a72-9369-018e882bff72","Type":"ContainerStarted","Data":"020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7"}
Oct 29 22:52:26.481782 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.481402    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" event={"ID":"52ad91ce-c59e-4a72-9369-018e882bff72","Type":"ContainerStarted","Data":"f1d6b1634bae0e140654718c49d833aff117bcce37f6514d36efae63ff56e7fe"}
Oct 29 22:52:26.482899 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.482873    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" event={"ID":"68408cd1-eda7-4df5-80ad-e82f9cf809e9","Type":"ContainerStarted","Data":"f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796"}
Oct 29 22:52:26.483030 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.482917    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" event={"ID":"68408cd1-eda7-4df5-80ad-e82f9cf809e9","Type":"ContainerStarted","Data":"1db358bb08fcea0f62140463cd8bc0e346b3bd2e9ad961bb63c3b1a1c64963b6"}
Oct 29 22:52:26.484546 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.484518    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" event={"ID":"a41b48a5-2064-4e12-87b5-4c8b3fb8694c","Type":"ContainerStarted","Data":"7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862"}
Oct 29 22:52:26.484653 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.484569    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" event={"ID":"a41b48a5-2064-4e12-87b5-4c8b3fb8694c","Type":"ContainerStarted","Data":"99a072f8d260b5fb185b6e4f2599585fa4eb1808fa18707375b72c7aa8f05c11"}
Oct 29 22:52:26.527687 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.527634    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podStartSLOduration=1.5276202749999999 podStartE2EDuration="1.527620275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:26.526305329 +0000 UTC m=+603.629381615" watchObservedRunningTime="2026-10-29 22:52:26.527620275 +0000 UTC m=+603.630696543"
Oct 29 22:52:26.571682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.571653    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:26.574580 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.574530    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podStartSLOduration=2.574515613 podStartE2EDuration="2.574515613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:26.572662876 +0000 UTC m=+603.675739159" watchObservedRunningTime="2026-10-29 22:52:26.574515613 +0000 UTC m=+603.677591889"
Oct 29 22:52:26.577170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.577132    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:26.620556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.620506    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podStartSLOduration=1.62049287 podStartE2EDuration="1.62049287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:26.619103354 +0000 UTC m=+603.722179626" watchObservedRunningTime="2026-10-29 22:52:26.62049287 +0000 UTC m=+603.723569141"
Oct 29 22:52:26.861772 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.861701    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:26.985909 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:26.985880    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:27.277453 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.277420    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:27.282508 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.282482    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:27.489367 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.489331    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:27.489942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.489660    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:27.489942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.489703    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:27.490407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.490384    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:27.490527 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.490435    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:52:27.490917 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.490887    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:27.867111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.867083    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:27.979592 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:27.979560    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:28.129888 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:28.129811    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:28.495059 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:28.494965    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:28.496088 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:28.496070    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:28.785690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:28.785650    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:28.885503 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:28.885468    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.009957 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:29.009917    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.144183 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:29.144085    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.272274 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:29.272237    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:29.834762 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:29.834734    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:30.162858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.162774    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"]
Oct 29 22:52:30.166429 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.166412    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.182336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.182306    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:30.188965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.188937    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-20f88f\""
Oct 29 22:52:30.194667 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.194641    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"]
Oct 29 22:52:30.203274 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.203250    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-20f88f-dockercfg-hw4mb\""
Oct 29 22:52:30.228678 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.228644    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxvm\" (UniqueName: \"kubernetes.io/projected/5943fc94-a232-47b8-9efb-01b669ed2787-kube-api-access-xsxvm\") pod \"payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.228678 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.228691    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943fc94-a232-47b8-9efb-01b669ed2787-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.330223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.330181    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxvm\" (UniqueName: \"kubernetes.io/projected/5943fc94-a232-47b8-9efb-01b669ed2787-kube-api-access-xsxvm\") pod \"payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.330223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.330221    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943fc94-a232-47b8-9efb-01b669ed2787-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.330976 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.330955    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943fc94-a232-47b8-9efb-01b669ed2787-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.389348 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.389266    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxvm\" (UniqueName: \"kubernetes.io/projected/5943fc94-a232-47b8-9efb-01b669ed2787-kube-api-access-xsxvm\") pod \"payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.392275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.392127    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:30.481738 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.481653    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:30.541670 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.541629    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:30.783317 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:30.783279    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"]
Oct 29 22:52:30.783423 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:30.783277    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5943fc94_a232_47b8_9efb_01b669ed2787.slice/crio-676d13e4d56ed5a12094ee79e1e63d22c01ec01162a3abb6a784357a05ab6fae WatchSource:0}: Error finding container 676d13e4d56ed5a12094ee79e1e63d22c01ec01162a3abb6a784357a05ab6fae: Status 404 returned error can't find the container with id 676d13e4d56ed5a12094ee79e1e63d22c01ec01162a3abb6a784357a05ab6fae
Oct 29 22:52:31.510687 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:31.510654    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" event={"ID":"5943fc94-a232-47b8-9efb-01b669ed2787","Type":"ContainerStarted","Data":"94a6a56bf5c5c49c09c671dcf701dbb80ea61b523b9dcbca4fc368b91cf9b978"}
Oct 29 22:52:31.511082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:31.510703    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" event={"ID":"5943fc94-a232-47b8-9efb-01b669ed2787","Type":"ContainerStarted","Data":"676d13e4d56ed5a12094ee79e1e63d22c01ec01162a3abb6a784357a05ab6fae"}
Oct 29 22:52:31.511082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:31.510759    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:31.556889 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:31.556839    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" podStartSLOduration=1.556826555 podStartE2EDuration="1.556826555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:31.555434041 +0000 UTC m=+608.658510314" watchObservedRunningTime="2026-10-29 22:52:31.556826555 +0000 UTC m=+608.659902826"
Oct 29 22:52:31.732537 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:31.732493    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:31.839329 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:31.839301    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:31.983659 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:31.983628    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.791968 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:32.791928    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.913537 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:32.913505    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:32.960226 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:32.960193    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"]
Oct 29 22:52:32.963655 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:32.963636    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:32.987456 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:32.987428    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"]
Oct 29 22:52:33.019519 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.019491    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-5bf55a\""
Oct 29 22:52:33.032455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.032433    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-5bf55a-dockercfg-wztfx\""
Oct 29 22:52:33.059609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.059539    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxsgr\" (UniqueName: \"kubernetes.io/projected/db34ef8b-313c-4614-b15d-09a6ac3dbe63-kube-api-access-bxsgr\") pod \"payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:33.059739 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.059617    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34ef8b-313c-4614-b15d-09a6ac3dbe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:33.160599 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.160565    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxsgr\" (UniqueName: \"kubernetes.io/projected/db34ef8b-313c-4614-b15d-09a6ac3dbe63-kube-api-access-bxsgr\") pod \"payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:33.160823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.160631    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34ef8b-313c-4614-b15d-09a6ac3dbe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:33.161352 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.161325    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34ef8b-313c-4614-b15d-09a6ac3dbe63-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:33.182273 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.182247    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxsgr\" (UniqueName: \"kubernetes.io/projected/db34ef8b-313c-4614-b15d-09a6ac3dbe63-kube-api-access-bxsgr\") pod \"payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:33.239627 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:52:33.239592    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:52:33.276652 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.276622    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:33.627836 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:33.627807    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb34ef8b_313c_4614_b15d_09a6ac3dbe63.slice/crio-ace517bf5b3a52b44563ee959bcdea230748ba00cbfb5647ea6992df52012b37 WatchSource:0}: Error finding container ace517bf5b3a52b44563ee959bcdea230748ba00cbfb5647ea6992df52012b37: Status 404 returned error can't find the container with id ace517bf5b3a52b44563ee959bcdea230748ba00cbfb5647ea6992df52012b37
Oct 29 22:52:33.628455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.628428    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"]
Oct 29 22:52:33.765875 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:33.765750    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:34.526110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:34.526077    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" event={"ID":"db34ef8b-313c-4614-b15d-09a6ac3dbe63","Type":"ContainerStarted","Data":"53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0"}
Oct 29 22:52:34.526557 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:34.526126    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" event={"ID":"db34ef8b-313c-4614-b15d-09a6ac3dbe63","Type":"ContainerStarted","Data":"ace517bf5b3a52b44563ee959bcdea230748ba00cbfb5647ea6992df52012b37"}
Oct 29 22:52:34.526557 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:34.526180    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:34.571328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:34.571280    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" podStartSLOduration=2.57126697 podStartE2EDuration="2.57126697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:34.57001787 +0000 UTC m=+611.673094141" watchObservedRunningTime="2026-10-29 22:52:34.57126697 +0000 UTC m=+611.674343243"
Oct 29 22:52:35.089261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.089231    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:35.185326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.185264    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:35.206433 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.206393    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"]
Oct 29 22:52:35.209854 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.209833    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.227422 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.227395    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"]
Oct 29 22:52:35.239417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.239393    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ffba57-dockercfg-m4rx5\""
Oct 29 22:52:35.239417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.239397    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ffba57\""
Oct 29 22:52:35.275189 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.275156    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"]
Oct 29 22:52:35.278550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.278526    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.280990 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.280968    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adab7618-52c4-4106-a1f2-76d19ab11656-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.281111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.281041    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w67b9\" (UniqueName: \"kubernetes.io/projected/adab7618-52c4-4106-a1f2-76d19ab11656-kube-api-access-w67b9\") pod \"payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.304519 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.304484    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"]
Oct 29 22:52:35.363450 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.363379    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:35.381705 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.381674    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w67b9\" (UniqueName: \"kubernetes.io/projected/adab7618-52c4-4106-a1f2-76d19ab11656-kube-api-access-w67b9\") pod \"payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.381883 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.381715    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9904b506-4dc7-44a9-a5a3-bb413392aa03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.381883 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.381789    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adab7618-52c4-4106-a1f2-76d19ab11656-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.382010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.381909    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8457\" (UniqueName: \"kubernetes.io/projected/9904b506-4dc7-44a9-a5a3-bb413392aa03-kube-api-access-b8457\") pod \"payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.382538 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.382518    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adab7618-52c4-4106-a1f2-76d19ab11656-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.403395 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.403367    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w67b9\" (UniqueName: \"kubernetes.io/projected/adab7618-52c4-4106-a1f2-76d19ab11656-kube-api-access-w67b9\") pod \"payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.464831 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.464801    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:35.482395 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.482367    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9904b506-4dc7-44a9-a5a3-bb413392aa03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.482556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.482435    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8457\" (UniqueName: \"kubernetes.io/projected/9904b506-4dc7-44a9-a5a3-bb413392aa03-kube-api-access-b8457\") pod \"payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.483100 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.483079    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9904b506-4dc7-44a9-a5a3-bb413392aa03-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.504810 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.504777    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8457\" (UniqueName: \"kubernetes.io/projected/9904b506-4dc7-44a9-a5a3-bb413392aa03-kube-api-access-b8457\") pod \"payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.522092 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.522064    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:35.574615 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.574527    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:35.600539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.599820    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:35.842734 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:35.842698    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadab7618_52c4_4106_a1f2_76d19ab11656.slice/crio-b66d63da7e4fb3871e9280e23708fc5fa6a1752a8373487c0c7bd68082633dca WatchSource:0}: Error finding container b66d63da7e4fb3871e9280e23708fc5fa6a1752a8373487c0c7bd68082633dca: Status 404 returned error can't find the container with id b66d63da7e4fb3871e9280e23708fc5fa6a1752a8373487c0c7bd68082633dca
Oct 29 22:52:35.843029 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.843011    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"]
Oct 29 22:52:35.967752 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:35.967719    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"]
Oct 29 22:52:35.968140 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:35.968120    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9904b506_4dc7_44a9_a5a3_bb413392aa03.slice/crio-43b43a14beae010e4acf2dd04e455d0a60f417cde26ee71f6bc2a3432b02e445 WatchSource:0}: Error finding container 43b43a14beae010e4acf2dd04e455d0a60f417cde26ee71f6bc2a3432b02e445: Status 404 returned error can't find the container with id 43b43a14beae010e4acf2dd04e455d0a60f417cde26ee71f6bc2a3432b02e445
Oct 29 22:52:36.539732 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.539690    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" event={"ID":"9904b506-4dc7-44a9-a5a3-bb413392aa03","Type":"ContainerStarted","Data":"8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7"}
Oct 29 22:52:36.539936 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.539779    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" event={"ID":"9904b506-4dc7-44a9-a5a3-bb413392aa03","Type":"ContainerStarted","Data":"43b43a14beae010e4acf2dd04e455d0a60f417cde26ee71f6bc2a3432b02e445"}
Oct 29 22:52:36.539936 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.539809    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:36.541393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.541367    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" event={"ID":"adab7618-52c4-4106-a1f2-76d19ab11656","Type":"ContainerStarted","Data":"99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807"}
Oct 29 22:52:36.541504 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.541406    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" event={"ID":"adab7618-52c4-4106-a1f2-76d19ab11656","Type":"ContainerStarted","Data":"b66d63da7e4fb3871e9280e23708fc5fa6a1752a8373487c0c7bd68082633dca"}
Oct 29 22:52:36.541504 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.541452    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:36.640723 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.640672    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" podStartSLOduration=1.640656324 podStartE2EDuration="1.640656324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:36.640444312 +0000 UTC m=+613.743520585" watchObservedRunningTime="2026-10-29 22:52:36.640656324 +0000 UTC m=+613.743732594"
Oct 29 22:52:36.641457 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.641428    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" podStartSLOduration=1.64141756 podStartE2EDuration="1.64141756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:36.587411197 +0000 UTC m=+613.690487480" watchObservedRunningTime="2026-10-29 22:52:36.64141756 +0000 UTC m=+613.744493831"
Oct 29 22:52:36.795355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.795212    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:36.910128 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.910096    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:36.963073 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.963043    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"]
Oct 29 22:52:36.966628 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.966613    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:36.985455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.985247    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-81f92d-dockercfg-74697\""
Oct 29 22:52:36.985455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.985392    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-81f92d\""
Oct 29 22:52:36.987124 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:36.987091    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"]
Oct 29 22:52:37.012178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.008758    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28h9\" (UniqueName: \"kubernetes.io/projected/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-kube-api-access-k28h9\") pod \"payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:37.012178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.008863    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:37.118254 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.110649    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28h9\" (UniqueName: \"kubernetes.io/projected/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-kube-api-access-k28h9\") pod \"payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:37.118254 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.110743    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:37.118254 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.113410    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:37.139251 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.139223    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28h9\" (UniqueName: \"kubernetes.io/projected/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-kube-api-access-k28h9\") pod \"payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:37.221577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.221544    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:37.302536 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.302491    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:37.337866 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.337526    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:37.455588 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.455560    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:37.627688 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:37.627167    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b24ac3e_d12b_41fb_b648_a97fb3ccdc3d.slice/crio-fefc84466feb334c271de96480d45f0e740758436a23562d9f8df61a262e8267 WatchSource:0}: Error finding container fefc84466feb334c271de96480d45f0e740758436a23562d9f8df61a262e8267: Status 404 returned error can't find the container with id fefc84466feb334c271de96480d45f0e740758436a23562d9f8df61a262e8267
Oct 29 22:52:37.627688 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.627260    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"]
Oct 29 22:52:37.696681 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.696645    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:37.796780 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:37.796654    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:38.554489 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:38.554423    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" event={"ID":"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d","Type":"ContainerStarted","Data":"3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7"}
Oct 29 22:52:38.554489 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:38.554509    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" event={"ID":"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d","Type":"ContainerStarted","Data":"fefc84466feb334c271de96480d45f0e740758436a23562d9f8df61a262e8267"}
Oct 29 22:52:38.555390 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:38.555369    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:38.624659 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:38.619263    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" podStartSLOduration=2.619130974 podStartE2EDuration="2.619130974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:38.606701454 +0000 UTC m=+615.709777727" watchObservedRunningTime="2026-10-29 22:52:38.619130974 +0000 UTC m=+615.722207246"
Oct 29 22:52:38.855382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:38.855290    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:38.958804 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:38.957530    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:39.053551 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.053520    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:39.328539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.328505    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:39.436882 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.436850    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:39.511214 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.511179    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"]
Oct 29 22:52:39.514491 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.514473    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.531665 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.531638    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-a2e7c8\""
Oct 29 22:52:39.545379 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.545348    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-a2e7c8-dockercfg-krp9s\""
Oct 29 22:52:39.547428 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.547401    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"]
Oct 29 22:52:39.562872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.562122    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8454cc6-c9b0-4a21-a803-7088233ace04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.562872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.562227    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw4nh\" (UniqueName: \"kubernetes.io/projected/a8454cc6-c9b0-4a21-a803-7088233ace04-kube-api-access-kw4nh\") pod \"payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.597249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.596358    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"]
Oct 29 22:52:39.600247 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.600225    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:39.618242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.618213    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"]
Oct 29 22:52:39.686177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.685602    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw4nh\" (UniqueName: \"kubernetes.io/projected/a8454cc6-c9b0-4a21-a803-7088233ace04-kube-api-access-kw4nh\") pod \"payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.686177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.685699    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8454cc6-c9b0-4a21-a803-7088233ace04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.686177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.685749    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx7rk\" (UniqueName: \"kubernetes.io/projected/a48caf36-726a-4a6a-8dbb-990fb2c03bff-kube-api-access-hx7rk\") pod \"payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:39.686177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.685886    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48caf36-726a-4a6a-8dbb-990fb2c03bff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:39.688125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.687570    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8454cc6-c9b0-4a21-a803-7088233ace04-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.737181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.732402    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw4nh\" (UniqueName: \"kubernetes.io/projected/a8454cc6-c9b0-4a21-a803-7088233ace04-kube-api-access-kw4nh\") pod \"payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.793356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.792706    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48caf36-726a-4a6a-8dbb-990fb2c03bff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:39.793356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.792837    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx7rk\" (UniqueName: \"kubernetes.io/projected/a48caf36-726a-4a6a-8dbb-990fb2c03bff-kube-api-access-hx7rk\") pod \"payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:39.794085 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.794055    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48caf36-726a-4a6a-8dbb-990fb2c03bff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:39.817876 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.817839    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx7rk\" (UniqueName: \"kubernetes.io/projected/a48caf36-726a-4a6a-8dbb-990fb2c03bff-kube-api-access-hx7rk\") pod \"payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:39.827242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.827209    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:39.914389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:39.914015    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:40.145097 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.145071    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"]
Oct 29 22:52:40.145422 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:40.145401    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8454cc6_c9b0_4a21_a803_7088233ace04.slice/crio-ebebc9dced772a42a9c9a4bc8168e9368455f134ee8e0c396764678516e58ac3 WatchSource:0}: Error finding container ebebc9dced772a42a9c9a4bc8168e9368455f134ee8e0c396764678516e58ac3: Status 404 returned error can't find the container with id ebebc9dced772a42a9c9a4bc8168e9368455f134ee8e0c396764678516e58ac3
Oct 29 22:52:40.242724 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:40.242697    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48caf36_726a_4a6a_8dbb_990fb2c03bff.slice/crio-007414e9c8c0a333ee0d22626d6d72ba50ccb539c7307ce4771f1656df4c3de1 WatchSource:0}: Error finding container 007414e9c8c0a333ee0d22626d6d72ba50ccb539c7307ce4771f1656df4c3de1: Status 404 returned error can't find the container with id 007414e9c8c0a333ee0d22626d6d72ba50ccb539c7307ce4771f1656df4c3de1
Oct 29 22:52:40.242902 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.242882    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"]
Oct 29 22:52:40.576184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.575813    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" event={"ID":"a8454cc6-c9b0-4a21-a803-7088233ace04","Type":"ContainerStarted","Data":"cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e"}
Oct 29 22:52:40.576184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.575889    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" event={"ID":"a8454cc6-c9b0-4a21-a803-7088233ace04","Type":"ContainerStarted","Data":"ebebc9dced772a42a9c9a4bc8168e9368455f134ee8e0c396764678516e58ac3"}
Oct 29 22:52:40.580169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.576789    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:40.584171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.580538    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" event={"ID":"a48caf36-726a-4a6a-8dbb-990fb2c03bff","Type":"ContainerStarted","Data":"047c57715356baaf3fea68214c81703c1b931e141a9b1060ae6f8b4a94ff1a56"}
Oct 29 22:52:40.589172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.588568    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" event={"ID":"a48caf36-726a-4a6a-8dbb-990fb2c03bff","Type":"ContainerStarted","Data":"007414e9c8c0a333ee0d22626d6d72ba50ccb539c7307ce4771f1656df4c3de1"}
Oct 29 22:52:40.589172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.588818    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:40.644504 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.644451    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" podStartSLOduration=1.644437631 podStartE2EDuration="1.644437631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:40.641597254 +0000 UTC m=+617.744673524" watchObservedRunningTime="2026-10-29 22:52:40.644437631 +0000 UTC m=+617.747513902"
Oct 29 22:52:40.695172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.694875    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" podStartSLOduration=1.694843857 podStartE2EDuration="1.694843857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:40.693655315 +0000 UTC m=+617.796731587" watchObservedRunningTime="2026-10-29 22:52:40.694843857 +0000 UTC m=+617.797920135"
Oct 29 22:52:40.695172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.694932    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:40.803317 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:40.803276    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:42.396174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:42.396008    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:42.498173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:42.497716    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:42.545168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:42.544991    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 22:52:42.633218 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:42.633115    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:43.540364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.540301    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:52:43.540801 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.540414    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:52:43.863933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.863854    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"]
Oct 29 22:52:43.867744 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.867720    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:43.915173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.914705    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l5lq\""
Oct 29 22:52:43.919177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.915395    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-ffba57-serving-cert\""
Oct 29 22:52:43.919177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.915477    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 22:52:43.919177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.915578    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 22:52:43.919177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.915620    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 22:52:43.928818 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.928575    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"]
Oct 29 22:52:43.931888 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.931844    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:43.957463 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.957433    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:43.957622 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.957494    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed11e89-0ac8-4cbf-b997-1a35e785af7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:43.957673 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.957630    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-metrics-certs\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:43.957813 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:43.957790    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrpj\" (UniqueName: \"kubernetes.io/projected/eed11e89-0ac8-4cbf-b997-1a35e785af7f-kube-api-access-jzrpj\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.031931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.031896    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:44.058928 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.058896    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-metrics-certs\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.059391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.059349    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrpj\" (UniqueName: \"kubernetes.io/projected/eed11e89-0ac8-4cbf-b997-1a35e785af7f-kube-api-access-jzrpj\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.059552 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.059533    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.059631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.059581    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed11e89-0ac8-4cbf-b997-1a35e785af7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.060259 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.060234    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed11e89-0ac8-4cbf-b997-1a35e785af7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.061940 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.061900    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.063500 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.063477    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-metrics-certs\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.083954 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.083923    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrpj\" (UniqueName: \"kubernetes.io/projected/eed11e89-0ac8-4cbf-b997-1a35e785af7f-kube-api-access-jzrpj\") pod \"maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.154984 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.154826    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:44.245217 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.245182    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:44.621705 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.621596    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"]
Oct 29 22:52:44.625933 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:52:44.625888    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed11e89_0ac8_4cbf_b997_1a35e785af7f.slice/crio-9949d1aa2f0e0533781d5d66061e20d326d6e3691f1299cb655e2c668d1f2c14 WatchSource:0}: Error finding container 9949d1aa2f0e0533781d5d66061e20d326d6e3691f1299cb655e2c668d1f2c14: Status 404 returned error can't find the container with id 9949d1aa2f0e0533781d5d66061e20d326d6e3691f1299cb655e2c668d1f2c14
Oct 29 22:52:44.644857 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:44.644824    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp" event={"ID":"eed11e89-0ac8-4cbf-b997-1a35e785af7f","Type":"ContainerStarted","Data":"9949d1aa2f0e0533781d5d66061e20d326d6e3691f1299cb655e2c668d1f2c14"}
Oct 29 22:52:45.537627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:45.537593    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 22:52:45.919224 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:45.913735    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:47.550235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:47.550105    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 22:52:47.550845 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:47.550816    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 22:52:47.635681 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:47.635647    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:48.706732 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:48.706687    4839 generic.go:356] "Generic (PLEG): container finished" podID="11c17881-8e31-4bc4-a55e-6005c5225912" containerID="61e15854a4450b37531bc037bf56e680feff016aa0027daa1088529d478863bd" exitCode=0
Oct 29 22:52:48.707210 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:48.706814    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" event={"ID":"11c17881-8e31-4bc4-a55e-6005c5225912","Type":"ContainerDied","Data":"61e15854a4450b37531bc037bf56e680feff016aa0027daa1088529d478863bd"}
Oct 29 22:52:48.707326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:48.707303    4839 scope.go:122] "RemoveContainer" containerID="61e15854a4450b37531bc037bf56e680feff016aa0027daa1088529d478863bd"
Oct 29 22:52:48.709213 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:48.709177    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:48.709360 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:48.709274    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:48.709360 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:48.709334    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:49.410172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.409751    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:49.729870 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.729786    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" event={"ID":"11c17881-8e31-4bc4-a55e-6005c5225912","Type":"ContainerStarted","Data":"ca7b1b7684c70d93401669d78986137a24fabecae00c37f00222be2076b88d40"}
Oct 29 22:52:49.732270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.732240    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp" event={"ID":"eed11e89-0ac8-4cbf-b997-1a35e785af7f","Type":"ContainerStarted","Data":"22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08"}
Oct 29 22:52:49.732816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.732797    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:49.899677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.899600    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp" podStartSLOduration=2.10582532 podStartE2EDuration="6.899581843s" totalImagesPullingTime="4.793756523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:52:49.898188377 +0000 UTC m=+627.001264649" watchObservedRunningTime="2026-10-29 22:52:49.899581843 +0000 UTC m=+627.002658116"
Oct 29 22:52:49.990302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.987565    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:49.990302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.987701    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:52:49.990302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:49.987747    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 29 22:52:50.610326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:50.609332    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 22:52:50.987228 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:50.984579    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:52:50.987228 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:50.984646    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 29 22:52:51.304670 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:51.304601    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:51.986172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:51.982813    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:52:51.986172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:51.982888    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 29 22:52:52.635772 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:52.635738    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 22:52:52.636172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:52.636037    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 22:52:52.998227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:52.993763    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:52:52.998227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:52.993820    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 29 22:52:52.998227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:52.994568    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:53.134474 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.134434    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:53.766351 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.766317    4839 generic.go:356] "Generic (PLEG): container finished" podID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerID="7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862" exitCode=0
Oct 29 22:52:53.766806 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.766413    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" event={"ID":"a41b48a5-2064-4e12-87b5-4c8b3fb8694c","Type":"ContainerDied","Data":"7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862"}
Oct 29 22:52:53.766806 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.766759    4839 scope.go:122] "RemoveContainer" containerID="7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862"
Oct 29 22:52:53.769641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.768428    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:53.769641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.768519    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:53.769641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.768567    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:53.796223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.794984    4839 generic.go:356] "Generic (PLEG): container finished" podID="52ad91ce-c59e-4a72-9369-018e882bff72" containerID="020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7" exitCode=0
Oct 29 22:52:53.796223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.795056    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" event={"ID":"52ad91ce-c59e-4a72-9369-018e882bff72","Type":"ContainerDied","Data":"020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7"}
Oct 29 22:52:53.796223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.795541    4839 scope.go:122] "RemoveContainer" containerID="020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7"
Oct 29 22:52:53.799809 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.799774    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:53.799911 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.799863    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:53.799951 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.799912    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:53.922054 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.921934    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:53.986174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.985557    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:52:53.986174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:53.985620    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 29 22:52:54.023533 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.023451    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:54.328658 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.328530    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:54.460997 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.460959    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:54.840173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.831085    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" event={"ID":"52ad91ce-c59e-4a72-9369-018e882bff72","Type":"ContainerStarted","Data":"be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2"}
Oct 29 22:52:54.868165 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.863828    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:54.868165 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.867371    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:52:54.868165 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.867424    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 29 22:52:54.894052 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.892879    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" event={"ID":"a41b48a5-2064-4e12-87b5-4c8b3fb8694c","Type":"ContainerStarted","Data":"43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b"}
Oct 29 22:52:54.931006 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.930970    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:54.993321 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.993266    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 29 22:52:54.993615 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:54.993586    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 29 22:52:55.065814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:55.065761    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:55.163094 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:55.163015    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:55.279756 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:55.279722    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:55.280106 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:55.280081    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:55.280305 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:55.280278    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:52:55.862514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:55.862476    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:52:55.862908 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:55.862536    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 29 22:52:56.020081 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.020047    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:56.021110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.021072    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:56.021601 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.021580    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:56.024182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.023890    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 22:52:56.331173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.281546    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:56.331173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.281607    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:52:56.759425 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.759338    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 22:52:56.862274 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.862234    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:52:56.862459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.862303    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 29 22:52:56.957010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:56.956949    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:57.279089 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:57.279050    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:57.279353 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:57.279324    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:52:57.503808 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:57.503775    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:52:57.862528 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:57.862176    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:52:57.862528 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:57.862230    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 29 22:52:58.280629 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.280582    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:58.281300 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.281249    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:52:58.541238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.539365    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:52:58.592978 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.592941    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:52:58.593351 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.593317    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:52:58.648617 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.648550    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:58.862125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.862047    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:52:58.862125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:58.862100    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 29 22:52:59.058194 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.058161    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:59.167773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.167690    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:59.278170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.277539    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.278170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.277647    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:52:59.514773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.514620    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:59.737895 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.732091    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:52:59.861896 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.861857    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Oct 29 22:52:59.862071 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.861916    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Oct 29 22:52:59.947197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.946757    4839 generic.go:356] "Generic (PLEG): container finished" podID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerID="f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796" exitCode=0
Oct 29 22:52:59.947197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.946855    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" event={"ID":"68408cd1-eda7-4df5-80ad-e82f9cf809e9","Type":"ContainerDied","Data":"f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796"}
Oct 29 22:52:59.947407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.947222    4839 scope.go:122] "RemoveContainer" containerID="f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796"
Oct 29 22:52:59.948559 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.948533    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:59.948610 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.948600    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:59.948643 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.948634    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:52:59.988168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:52:59.982823    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:00.158262 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.134717    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:00.277655 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.277621    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Oct 29 22:53:00.277945 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.277904    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Oct 29 22:53:00.300275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.300018    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:00.497633 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.497541    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:00.866950 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.866918    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:53:00.867960 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.867937    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:53:00.940080 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.940044    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:00.962534 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:00.962388    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" event={"ID":"68408cd1-eda7-4df5-80ad-e82f9cf809e9","Type":"ContainerStarted","Data":"683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03"}
Oct 29 22:53:01.295334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:01.295298    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:53:01.296494 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:01.296468    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 22:53:01.579252 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:01.577569    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:53:01.579252 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:01.577698    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:01.579252 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:01.577741    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 29 22:53:02.577624 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:02.577582    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:02.578277 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:02.578225    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 29 22:53:02.894548 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:02.894450    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:03.237859 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:53:03.237775    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:53:03.580171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:03.578621    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:03.580171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:03.578685    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 29 22:53:04.366803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.366767    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:04.492889 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.492636    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:04.593170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.579519    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:04.593170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.579576    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 29 22:53:04.654346 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.654269    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"]
Oct 29 22:53:04.658037 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.658012    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.700303 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.700253    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"]
Oct 29 22:53:04.711566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.711527    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"default-dockercfg-nmcsr\""
Oct 29 22:53:04.711751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.711628    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"kube-root-ca.crt\""
Oct 29 22:53:04.711751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.711629    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"simulator-w6-81f92d-kserve-self-signed-certs\""
Oct 29 22:53:04.711751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.711696    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"openshift-service-ca.crt\""
Oct 29 22:53:04.719754 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.719732    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:04.737408 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.737379    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-model-cache\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.737539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.737412    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tls-certs\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.737539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.737434    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tmp-dir\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.737539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.737523    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzj7\" (UniqueName: \"kubernetes.io/projected/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-kube-api-access-hhzj7\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.737640 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.737589    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-home\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.737673 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.737641    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-dshm\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838404 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838364    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-model-cache\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838404 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838408    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tls-certs\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838437    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tmp-dir\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838471    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzj7\" (UniqueName: \"kubernetes.io/projected/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-kube-api-access-hhzj7\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838524    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-home\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838585    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-dshm\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838857 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838834    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-model-cache\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838928 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838866    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tmp-dir\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.838986 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.838951    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-home\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.840743 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.840725    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-dshm\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.841216 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.841197    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tls-certs\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.859771 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.859740    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzj7\" (UniqueName: \"kubernetes.io/projected/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-kube-api-access-hhzj7\") pod \"simulator-w6-81f92d-kserve-6684576b69-99t7r\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") " pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:04.994663 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:04.994519    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:05.000564 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:05.000536    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:05.111967 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:05.111932    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:05.224568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:05.224531    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:05.312428 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:05.312403    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"]
Oct 29 22:53:05.313120 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:53:05.313096    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfda4363_0693_4a7e_8ea5_8babe3a0bfe3.slice/crio-86de31eb852a0bccb4a5a00ed868348d077d112d989c7383b2009da811d61668 WatchSource:0}: Error finding container 86de31eb852a0bccb4a5a00ed868348d077d112d989c7383b2009da811d61668: Status 404 returned error can't find the container with id 86de31eb852a0bccb4a5a00ed868348d077d112d989c7383b2009da811d61668
Oct 29 22:53:05.575009 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:05.574512    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Oct 29 22:53:05.575009 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:05.574573    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Oct 29 22:53:06.015295 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.015265    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" event={"ID":"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3","Type":"ContainerStarted","Data":"6154ad4f401294c8e5c56c63b7db2a838e06d04d26ee4a2c3351c328b7410dec"}
Oct 29 22:53:06.015683 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.015304    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" event={"ID":"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3","Type":"ContainerStarted","Data":"86de31eb852a0bccb4a5a00ed868348d077d112d989c7383b2009da811d61668"}
Oct 29 22:53:06.016824 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.016802    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:06.065473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.065414    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podStartSLOduration=2.065399992 podStartE2EDuration="2.065399992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:06.063727749 +0000 UTC m=+643.166804047" watchObservedRunningTime="2026-10-29 22:53:06.065399992 +0000 UTC m=+643.168476286"
Oct 29 22:53:06.366040 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.365948    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:06.574587 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.574556    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:53:06.574878 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.574857    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:53:06.575479 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:06.575457    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 22:53:07.000979 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:07.000945    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:07.626216 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:07.626183    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:08.170629 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.170600    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:08.831139 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.831104    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"]
Oct 29 22:53:08.834894 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.834867    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.852472 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.852445    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"]
Oct 29 22:53:08.864751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.864722    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5bf55a\"/\"openshift-service-ca.crt\""
Oct 29 22:53:08.864911 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.864744    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bf55a\"/\"default-dockercfg-4nfhm\""
Oct 29 22:53:08.867251 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.867230    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bf55a\"/\"simulator-w5-5bf55a-kserve-self-signed-certs\""
Oct 29 22:53:08.867564 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.867548    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5bf55a\"/\"kube-root-ca.crt\""
Oct 29 22:53:08.869382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.869343    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:08.882671 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.882636    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b386111-f2f1-4828-b667-ddd2b874184f-tls-certs\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.882827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.882693    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-model-cache\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.882827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.882777    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-home\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.882953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.882848    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvx6\" (UniqueName: \"kubernetes.io/projected/9b386111-f2f1-4828-b667-ddd2b874184f-kube-api-access-cjvx6\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.882953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.882898    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-dshm\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.882953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.882940    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-tmp-dir\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984277 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984244    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b386111-f2f1-4828-b667-ddd2b874184f-tls-certs\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984290    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-model-cache\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984380    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-home\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984426    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvx6\" (UniqueName: \"kubernetes.io/projected/9b386111-f2f1-4828-b667-ddd2b874184f-kube-api-access-cjvx6\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984456    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-dshm\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984493    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-tmp-dir\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984732    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-model-cache\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984765    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-home\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.984920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.984901    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-tmp-dir\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.986695 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.986680    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-dshm\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:08.987306 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:08.987290    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b386111-f2f1-4828-b667-ddd2b874184f-tls-certs\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:09.006103 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:09.006072    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvx6\" (UniqueName: \"kubernetes.io/projected/9b386111-f2f1-4828-b667-ddd2b874184f-kube-api-access-cjvx6\") pod \"simulator-w5-5bf55a-kserve-59df96f7d8-w888g\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") " pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:09.151082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:09.151001    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:09.472935 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:09.472900    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"]
Oct 29 22:53:09.838025 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:09.837990    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:10.037732 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:10.037691    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" event={"ID":"9b386111-f2f1-4828-b667-ddd2b874184f","Type":"ContainerStarted","Data":"195384b0297c02f3b460ebfaf40f258a8cd1d7cb428bbaec06839c21e36da927"}
Oct 29 22:53:10.037915 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:10.037743    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" event={"ID":"9b386111-f2f1-4828-b667-ddd2b874184f","Type":"ContainerStarted","Data":"0ffbbae1ee54fccf1a791bf3ad95b0505b47f6c9f1306e450450127db20cc6b0"}
Oct 29 22:53:10.084405 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:10.084355    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podStartSLOduration=2.084342619 podStartE2EDuration="2.084342619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:10.082310153 +0000 UTC m=+647.185386446" watchObservedRunningTime="2026-10-29 22:53:10.084342619 +0000 UTC m=+647.187418930"
Oct 29 22:53:10.151939 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:10.151858    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:11.565309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:11.565267    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:11.839611 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:11.839522    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:13.057419 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:13.057387    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:14.431775 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.431736    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"]
Oct 29 22:53:14.435837 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.435812    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.455717 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.455684    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"]
Oct 29 22:53:14.467680 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.467629    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"default-dockercfg-2mj5c\""
Oct 29 22:53:14.467865 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.467772    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"openshift-service-ca.crt\""
Oct 29 22:53:14.470272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.470245    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"simulator-w2-a2e7c8-kserve-self-signed-certs\""
Oct 29 22:53:14.470412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.470298    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"kube-root-ca.crt\""
Oct 29 22:53:14.486360 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.486334    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:14.544420 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.544383    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-home\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.544585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.544446    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-model-cache\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.544585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.544522    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-dshm\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.544585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.544563    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbp5c\" (UniqueName: \"kubernetes.io/projected/d2c81bda-33e7-482e-b302-806d09c7518c-kube-api-access-fbp5c\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.544686 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.544647    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-tmp-dir\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.544724 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.544684    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c81bda-33e7-482e-b302-806d09c7518c-tls-certs\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.587220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.587186    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:14.645468 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.645435    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-dshm\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.645468 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.645474    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbp5c\" (UniqueName: \"kubernetes.io/projected/d2c81bda-33e7-482e-b302-806d09c7518c-kube-api-access-fbp5c\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.645711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.645610    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-tmp-dir\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.645711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.645670    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c81bda-33e7-482e-b302-806d09c7518c-tls-certs\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.645782 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.645758    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-home\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.645885 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.645868    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-model-cache\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.646093 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.646061    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-tmp-dir\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.646266 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.646134    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-home\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.646266 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.646217    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-model-cache\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.647904 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.647882    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-dshm\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.648235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.648216    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c81bda-33e7-482e-b302-806d09c7518c-tls-certs\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.667264 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.667242    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbp5c\" (UniqueName: \"kubernetes.io/projected/d2c81bda-33e7-482e-b302-806d09c7518c-kube-api-access-fbp5c\") pod \"simulator-w2-a2e7c8-kserve-68d99944f-5d7dx\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") " pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:14.703225 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.701642    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:14.749810 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:14.749769    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:15.001081 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:15.000994    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:15.014214 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:15.014189    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:15.045765 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:15.045733    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"]
Oct 29 22:53:15.046127 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:53:15.046099    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2c81bda_33e7_482e_b302_806d09c7518c.slice/crio-3ea9ae24c4be2025326d451375ec2cd6e67f2366d1f5c61eddac51a79c480b76 WatchSource:0}: Error finding container 3ea9ae24c4be2025326d451375ec2cd6e67f2366d1f5c61eddac51a79c480b76: Status 404 returned error can't find the container with id 3ea9ae24c4be2025326d451375ec2cd6e67f2366d1f5c61eddac51a79c480b76
Oct 29 22:53:15.065135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:15.065101    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" event={"ID":"d2c81bda-33e7-482e-b302-806d09c7518c","Type":"ContainerStarted","Data":"3ea9ae24c4be2025326d451375ec2cd6e67f2366d1f5c61eddac51a79c480b76"}
Oct 29 22:53:15.076397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:15.076370    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:53:16.072392 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:16.072353    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" event={"ID":"d2c81bda-33e7-482e-b302-806d09c7518c","Type":"ContainerStarted","Data":"a094204d35153ea83365c6b9d5f8572601f4fd313138de2a33175dfb306cdff0"}
Oct 29 22:53:16.118650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:16.118599    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podStartSLOduration=2.118587198 podStartE2EDuration="2.118587198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:16.116732898 +0000 UTC m=+653.219809170" watchObservedRunningTime="2026-10-29 22:53:16.118587198 +0000 UTC m=+653.221663468"
Oct 29 22:53:16.294811 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:16.294778    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:16.749938 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:16.749898    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:17.929733 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:17.929699    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:18.804009 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:18.803978    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:18.909516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:18.909482    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:19.151910 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:19.151834    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:19.159668 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:19.159640    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:19.164994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:19.164972    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:19.687358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:19.687328    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:20.103724 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:20.103696    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:53:21.308202 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:21.308170    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:23.080389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:23.080359    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:24.750322 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:24.750285    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:24.762656 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:24.762625    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:24.773462 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:24.773432    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:25.131133 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:25.131099    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:53:26.279486 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:26.279452    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:27.883849 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:27.883801    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:29.360526 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:29.360494    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:31.042127 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.042089    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:31.540588 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.540553    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"]
Oct 29 22:53:31.546050 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.546022    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.559889 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.559863    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"]
Oct 29 22:53:31.571822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.571800    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"openshift-service-ca.crt\""
Oct 29 22:53:31.571822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.571810    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"kube-root-ca.crt\""
Oct 29 22:53:31.576358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.576331    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"default-dockercfg-l88xn\""
Oct 29 22:53:31.576451 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.576389    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-20f88f\"/\"premium-w1-20f88f-kserve-self-signed-certs\""
Oct 29 22:53:31.588516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.588494    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:31.621336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.621309    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-dshm\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.621482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.621344    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhvrt\" (UniqueName: \"kubernetes.io/projected/5397ab93-f411-494f-a840-5bfea3e38c39-kube-api-access-jhvrt\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.621482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.621471    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-tmp-dir\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.621584 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.621504    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-home\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.621646 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.621603    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5397ab93-f411-494f-a840-5bfea3e38c39-tls-certs\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.621694 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.621651    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-model-cache\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722349 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722318    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-tmp-dir\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722532 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722355    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-home\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722532 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722391    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5397ab93-f411-494f-a840-5bfea3e38c39-tls-certs\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722532 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722411    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-model-cache\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722532 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722461    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-dshm\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722762 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722600    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhvrt\" (UniqueName: \"kubernetes.io/projected/5397ab93-f411-494f-a840-5bfea3e38c39-kube-api-access-jhvrt\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722785    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-tmp-dir\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722892 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722868    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-home\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.722945 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.722872    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-model-cache\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.724703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.724674    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-dshm\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.725010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.724993    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5397ab93-f411-494f-a840-5bfea3e38c39-tls-certs\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.766915 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.766851    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhvrt\" (UniqueName: \"kubernetes.io/projected/5397ab93-f411-494f-a840-5bfea3e38c39-kube-api-access-jhvrt\") pod \"premium-w1-20f88f-kserve-7855b9b979-tq4gm\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") " pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:31.860526 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:31.860435    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:32.155436 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:53:32.155409    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5397ab93_f411_494f_a840_5bfea3e38c39.slice/crio-0af5a2526cd75437a073e62c6b63ece868b4e9522023c8c3d68745186684cc1b WatchSource:0}: Error finding container 0af5a2526cd75437a073e62c6b63ece868b4e9522023c8c3d68745186684cc1b: Status 404 returned error can't find the container with id 0af5a2526cd75437a073e62c6b63ece868b4e9522023c8c3d68745186684cc1b
Oct 29 22:53:32.156076 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:32.156055    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"]
Oct 29 22:53:32.615403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:32.615361    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:33.164967 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:33.164927    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" event={"ID":"5397ab93-f411-494f-a840-5bfea3e38c39","Type":"ContainerStarted","Data":"7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4"}
Oct 29 22:53:33.164967 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:33.164970    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" event={"ID":"5397ab93-f411-494f-a840-5bfea3e38c39","Type":"ContainerStarted","Data":"0af5a2526cd75437a073e62c6b63ece868b4e9522023c8c3d68745186684cc1b"}
Oct 29 22:53:33.210879 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:33.210822    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podStartSLOduration=2.2107977930000002 podStartE2EDuration="2.210797793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:33.209244829 +0000 UTC m=+670.312321103" watchObservedRunningTime="2026-10-29 22:53:33.210797793 +0000 UTC m=+670.313874064"
Oct 29 22:53:33.238872 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:53:33.238838    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:53:33.861590 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:33.861551    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:34.246883 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:34.246796    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:36.029243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:36.029100    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:37.547193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:37.547138    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:39.263234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:39.263198    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:39.400513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:39.400438    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:40.102521 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.102484    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"]
Oct 29 22:53:40.106632 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.106608    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.139324 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.139278    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5bf55a-openshift-default-dockercfg-hx7k2\""
Oct 29 22:53:40.141875 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.141844    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"]
Oct 29 22:53:40.213597 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.213565    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-podinfo\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.213785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.213621    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74110f0b-b412-42f2-884e-0438f0fc02a1-istiod-ca-cert\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.213785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.213761    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-token\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.213877 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.213849    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.213916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.213886    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-data\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.213970 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.213915    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-envoy\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.214026 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.213998    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b9dm\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-kube-api-access-5b9dm\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.214081 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.214041    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-certs\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.214132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.214108    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-credential-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315299    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-certs\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315333    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-credential-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315390    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-podinfo\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315494    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74110f0b-b412-42f2-884e-0438f0fc02a1-istiod-ca-cert\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315558    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-token\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315649    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315700    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-certs\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315709    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-data\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315760    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-credential-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.315869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315836    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-envoy\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.316249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315882    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b9dm\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-kube-api-access-5b9dm\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.316249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.315965    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-socket\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.316249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.316024    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-data\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.316383 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.316287    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74110f0b-b412-42f2-884e-0438f0fc02a1-istiod-ca-cert\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.317894 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.317824    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-podinfo\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.318017 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.317901    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-envoy\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.338368 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.338334    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-token\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.338746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.338723    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b9dm\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-kube-api-access-5b9dm\") pod \"e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") " pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.422802 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.422710    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:40.736237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.736208    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"]
Oct 29 22:53:40.736466 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:53:40.736442    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74110f0b_b412_42f2_884e_0438f0fc02a1.slice/crio-737d816448920e67c9d0d057767a68f74a9e7b8868185231c6dd0b47cdcff185 WatchSource:0}: Error finding container 737d816448920e67c9d0d057767a68f74a9e7b8868185231c6dd0b47cdcff185: Status 404 returned error can't find the container with id 737d816448920e67c9d0d057767a68f74a9e7b8868185231c6dd0b47cdcff185
Oct 29 22:53:40.738621 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.738586    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:53:40.738736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.738664    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:53:40.738736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.738706    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:53:40.958519 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:40.958487    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:41.216650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.216614    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" event={"ID":"74110f0b-b412-42f2-884e-0438f0fc02a1","Type":"ContainerStarted","Data":"da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f"}
Oct 29 22:53:41.216817 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.216678    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" event={"ID":"74110f0b-b412-42f2-884e-0438f0fc02a1","Type":"ContainerStarted","Data":"737d816448920e67c9d0d057767a68f74a9e7b8868185231c6dd0b47cdcff185"}
Oct 29 22:53:41.302113 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.302046    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podStartSLOduration=1.30202872 podStartE2EDuration="1.30202872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:41.286243239 +0000 UTC m=+678.389319537" watchObservedRunningTime="2026-10-29 22:53:41.30202872 +0000 UTC m=+678.405105084"
Oct 29 22:53:41.423495 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.423449    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:41.425117 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.425092    4839 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:53:41.425204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.425175    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 29 22:53:41.861352 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.861298    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:41.898649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:41.898608    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:42.232991 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:42.232905    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:53:42.423879 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:42.423844    4839 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:53:42.424297 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:42.423900    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 29 22:53:42.493244 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:42.492701    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:43.263640 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:43.263613    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:43.427253 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:43.426577    4839 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:53:43.427253 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:43.426644    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 29 22:53:44.194630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:44.194592    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:44.423899 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:44.423856    4839 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:53:44.424094 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:44.423925    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 29 22:53:45.423751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:45.423687    4839 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:53:45.424432 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:45.423765    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 29 22:53:46.064382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:46.064346    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:46.423942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:46.423851    4839 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 29 22:53:46.423942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:46.423908    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 29 22:53:47.427965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:47.427927    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:47.428384 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:47.428187    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:47.429004 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:47.428988    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 22:53:48.074082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.074048    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:48.796884 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.796844    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"]
Oct 29 22:53:48.802096 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.802066    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:48.820550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.820521    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"]
Oct 29 22:53:48.831770 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.831743    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"premium-w3-07eef2-kserve-self-signed-certs\""
Oct 29 22:53:48.831922 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.831850    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"default-dockercfg-r9mrg\""
Oct 29 22:53:48.859430 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.859402    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:48.880459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.880431    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"kube-root-ca.crt\""
Oct 29 22:53:48.880459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.880456    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-07eef2\"/\"openshift-service-ca.crt\""
Oct 29 22:53:48.942735 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.942697    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tmp-dir\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:48.942903 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.942751    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvlt\" (UniqueName: \"kubernetes.io/projected/cd1ec599-e3bc-40f4-bc65-ab513668d68b-kube-api-access-djvlt\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:48.942903 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.942873    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tls-certs\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:48.942983 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.942931    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-home\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:48.942983 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.942962    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-model-cache\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:48.943057 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:48.943002    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-dshm\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.043844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.043802    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tmp-dir\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.044162 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.044114    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvlt\" (UniqueName: \"kubernetes.io/projected/cd1ec599-e3bc-40f4-bc65-ab513668d68b-kube-api-access-djvlt\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.044336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.044316    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tls-certs\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.044494 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.044480    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-home\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.044609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.044595    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-model-cache\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.044722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.044707    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-dshm\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.046196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.046169    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tmp-dir\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.046923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.046868    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-home\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.047345 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.047317    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-model-cache\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.052033 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.052006    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tls-certs\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.054220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.052178    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-dshm\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.068412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.068385    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvlt\" (UniqueName: \"kubernetes.io/projected/cd1ec599-e3bc-40f4-bc65-ab513668d68b-kube-api-access-djvlt\") pod \"premium-w3-07eef2-kserve-75b97fcc66-j9xqf\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") " pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.117204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.117173    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:49.419666 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.419635    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"]
Oct 29 22:53:49.419794 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:53:49.419635    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1ec599_e3bc_40f4_bc65_ab513668d68b.slice/crio-3abe23d7b6cb6178149f828cda15f1a4a018a61aebf0b7d42e620464686f7f65 WatchSource:0}: Error finding container 3abe23d7b6cb6178149f828cda15f1a4a018a61aebf0b7d42e620464686f7f65: Status 404 returned error can't find the container with id 3abe23d7b6cb6178149f828cda15f1a4a018a61aebf0b7d42e620464686f7f65
Oct 29 22:53:49.785062 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:49.785031    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:50.269423 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:50.269383    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" event={"ID":"cd1ec599-e3bc-40f4-bc65-ab513668d68b","Type":"ContainerStarted","Data":"f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252"}
Oct 29 22:53:50.269796 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:50.269439    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" event={"ID":"cd1ec599-e3bc-40f4-bc65-ab513668d68b","Type":"ContainerStarted","Data":"3abe23d7b6cb6178149f828cda15f1a4a018a61aebf0b7d42e620464686f7f65"}
Oct 29 22:53:50.314690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:50.314621    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podStartSLOduration=2.314606714 podStartE2EDuration="2.314606714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:53:50.31350731 +0000 UTC m=+687.416583584" watchObservedRunningTime="2026-10-29 22:53:50.314606714 +0000 UTC m=+687.417682984"
Oct 29 22:53:50.316303 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:50.316278    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:51.117399 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:51.117365    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:51.466003 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:51.465904    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:52.946703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:52.946665    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:54.545331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:54.545296    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:56.061189 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:56.061132    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:56.474195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:56.474081    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:58.109782 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:58.109746    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:53:59.118214 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:59.118174    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:59.130408 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:59.130383    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:59.329958 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:59.329927    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:53:59.832351 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:53:59.832319    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:01.417521 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:01.417482    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:03.092938 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:03.092902    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:03.237415 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:54:03.237366    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:54:04.880965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:04.880935    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:06.780721 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:06.780683    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:08.412580 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:08.412543    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:10.064897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.064857    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:10.797255 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.797222    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"]
Oct 29 22:54:10.800915 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.800887    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.821558 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.821529    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"]
Oct 29 22:54:10.831959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.831924    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-07eef2-openshift-default-dockercfg-5l9dg\""
Oct 29 22:54:10.904088 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904051    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904408    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904564 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904534    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904623 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904601    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904668 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904656    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qn5b\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-kube-api-access-9qn5b\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904729    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904806 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904785    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904894 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904874    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:10.904937 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:10.904926    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006099 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006064    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006099 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006104    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006168    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006225    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006270    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006335    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006374    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006429    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qn5b\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-kube-api-access-9qn5b\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006458    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006784 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006676    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.006784 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006740    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.007009 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.006983    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.007141 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.007001    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.007515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.007494    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.008886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.008859    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.009242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.009220    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.031186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.031134    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qn5b\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-kube-api-access-9qn5b\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.032132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.032110    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.114254 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.114176    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:11.439683 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.439654    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"]
Oct 29 22:54:11.439798 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:11.439750    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad9d225_2e9b_4f14_96e7_2c297cee3ef4.slice/crio-185de4c738283f9eb1951951839a09d533f73d1779bcc2201df2b269d56f2607 WatchSource:0}: Error finding container 185de4c738283f9eb1951951839a09d533f73d1779bcc2201df2b269d56f2607: Status 404 returned error can't find the container with id 185de4c738283f9eb1951951839a09d533f73d1779bcc2201df2b269d56f2607
Oct 29 22:54:11.441924 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.441891    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:11.442011 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.441957    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:11.442011 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.441985    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:11.894042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:11.894004    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:12.405053 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:12.405011    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" event={"ID":"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4","Type":"ContainerStarted","Data":"0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5"}
Oct 29 22:54:12.405475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:12.405108    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" event={"ID":"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4","Type":"ContainerStarted","Data":"185de4c738283f9eb1951951839a09d533f73d1779bcc2201df2b269d56f2607"}
Oct 29 22:54:12.467291 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:12.467228    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podStartSLOduration=2.467216065 podStartE2EDuration="2.467216065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:12.46597676 +0000 UTC m=+709.569053054" watchObservedRunningTime="2026-10-29 22:54:12.467216065 +0000 UTC m=+709.570292335"
Oct 29 22:54:13.114470 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:13.114436    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:13.116036 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:13.116009    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 29 22:54:13.116194 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:13.116088    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 29 22:54:13.207322 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:13.207291    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:13.640505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:13.640475    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:14.115045 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.115012    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 29 22:54:14.115250 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.115090    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 29 22:54:14.613272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.613221    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:14.936436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.936352    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"]
Oct 29 22:54:14.940361 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.940337    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:14.956260 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.956232    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"]
Oct 29 22:54:14.967768 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.967734    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:14.967905 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:14.967824    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:54:15.054477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.054436    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.054477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.054477    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56tw\" (UniqueName: \"kubernetes.io/projected/101cba55-58fa-4cf9-b98d-2a27e4fcd797-kube-api-access-n56tw\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.054699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.054505    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.115330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.115298    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 29 22:54:15.115475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.115355    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 29 22:54:15.155366 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.155327    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.155366 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.155367    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56tw\" (UniqueName: \"kubernetes.io/projected/101cba55-58fa-4cf9-b98d-2a27e4fcd797-kube-api-access-n56tw\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.155620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.155393    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.156039 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.156005    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.157730 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.157707    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.193994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.193914    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56tw\" (UniqueName: \"kubernetes.io/projected/101cba55-58fa-4cf9-b98d-2a27e4fcd797-kube-api-access-n56tw\") pod \"maas-api-revoke-sub-05eb199b-kdt75\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.254391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.254357    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:15.517206 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.517175    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:15.606016 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:15.605885    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"]
Oct 29 22:54:15.629604 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:15.629567    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101cba55_58fa_4cf9_b98d_2a27e4fcd797.slice/crio-abb8ae3bd3f0b5b90ed71f4ece4e9681b7ab552e6fdd25fde05dabefcdfd572f WatchSource:0}: Error finding container abb8ae3bd3f0b5b90ed71f4ece4e9681b7ab552e6fdd25fde05dabefcdfd572f: Status 404 returned error can't find the container with id abb8ae3bd3f0b5b90ed71f4ece4e9681b7ab552e6fdd25fde05dabefcdfd572f
Oct 29 22:54:16.116276 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:16.116240    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 29 22:54:16.116663 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:16.116291    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 29 22:54:16.522185 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:16.521936    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75" event={"ID":"101cba55-58fa-4cf9-b98d-2a27e4fcd797","Type":"ContainerStarted","Data":"abb8ae3bd3f0b5b90ed71f4ece4e9681b7ab552e6fdd25fde05dabefcdfd572f"}
Oct 29 22:54:17.117040 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:17.116556    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 29 22:54:17.117040 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:17.116634    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 29 22:54:17.339860 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:17.339828    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:18.114776 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:18.114733    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Oct 29 22:54:18.114971 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:18.114803    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Oct 29 22:54:19.161513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:19.161479    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:19.444197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:19.444091    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:19.444541 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:19.444516    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:19.445897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:19.445867    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 22:54:19.543216 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:19.543182    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75" event={"ID":"101cba55-58fa-4cf9-b98d-2a27e4fcd797","Type":"ContainerStarted","Data":"06e3c6e42415c7098dcf049e2cc2998ac0ee660344e80d5ab5432174f246c215"}
Oct 29 22:54:19.598142 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:19.598067    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75" podStartSLOduration=2.474143084 podStartE2EDuration="5.598048228s" totalImagesPullingTime="3.123905144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:19.597058479 +0000 UTC m=+716.700134751" watchObservedRunningTime="2026-10-29 22:54:19.598048228 +0000 UTC m=+716.701124502"
Oct 29 22:54:20.551212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:20.551171    4839 generic.go:356] "Generic (PLEG): container finished" podID="101cba55-58fa-4cf9-b98d-2a27e4fcd797" containerID="06e3c6e42415c7098dcf049e2cc2998ac0ee660344e80d5ab5432174f246c215" exitCode=0
Oct 29 22:54:20.551723 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:20.551264    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75" event={"ID":"101cba55-58fa-4cf9-b98d-2a27e4fcd797","Type":"ContainerDied","Data":"06e3c6e42415c7098dcf049e2cc2998ac0ee660344e80d5ab5432174f246c215"}
Oct 29 22:54:20.774334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:20.774275    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:21.695128 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.695099    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:21.766023 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.765984    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101cba55-58fa-4cf9-b98d-2a27e4fcd797-kube-api-access-n56tw\" (UniqueName: \"kubernetes.io/projected/101cba55-58fa-4cf9-b98d-2a27e4fcd797-kube-api-access-n56tw\") pod \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") "
Oct 29 22:54:21.766240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.766099    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-openshift-ca\") pod \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") "
Oct 29 22:54:21.766240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.766121    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-xks-ca\") pod \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\" (UID: \"101cba55-58fa-4cf9-b98d-2a27e4fcd797\") "
Oct 29 22:54:21.766481 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.766452    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-openshift-ca" pod "101cba55-58fa-4cf9-b98d-2a27e4fcd797" (UID: "101cba55-58fa-4cf9-b98d-2a27e4fcd797"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:21.768419 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.768386    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-xks-ca" pod "101cba55-58fa-4cf9-b98d-2a27e4fcd797" (UID: "101cba55-58fa-4cf9-b98d-2a27e4fcd797"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:21.768512 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.768439    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101cba55-58fa-4cf9-b98d-2a27e4fcd797-kube-api-access-n56tw" pod "101cba55-58fa-4cf9-b98d-2a27e4fcd797" (UID: "101cba55-58fa-4cf9-b98d-2a27e4fcd797"). InnerVolumeSpecName "kube-api-access-n56tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:21.867405 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.867312    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56tw\" (UniqueName: \"kubernetes.io/projected/101cba55-58fa-4cf9-b98d-2a27e4fcd797-kube-api-access-n56tw\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:21.867405 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.867345    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:21.867405 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:21.867358    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101cba55-58fa-4cf9-b98d-2a27e4fcd797-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:22.563973 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:22.563943    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"
Oct 29 22:54:22.564282 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:22.563990    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75" event={"ID":"101cba55-58fa-4cf9-b98d-2a27e4fcd797","Type":"ContainerDied","Data":"abb8ae3bd3f0b5b90ed71f4ece4e9681b7ab552e6fdd25fde05dabefcdfd572f"}
Oct 29 22:54:22.564397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:22.564307    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abb8ae3bd3f0b5b90ed71f4ece4e9681b7ab552e6fdd25fde05dabefcdfd572f"
Oct 29 22:54:22.626718 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:22.626676    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:23.697436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.697393    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"]
Oct 29 22:54:23.698069 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.698048    4839 cpu_manager.go:414] "containerMap: removing container" podUID="101cba55-58fa-4cf9-b98d-2a27e4fcd797" containerName="revoke-keys"
Oct 29 22:54:23.698201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.698071    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101cba55-58fa-4cf9-b98d-2a27e4fcd797" containerName="revoke-keys"
Oct 29 22:54:23.698270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.698212    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="101cba55-58fa-4cf9-b98d-2a27e4fcd797" containerName="revoke-keys"
Oct 29 22:54:23.702696 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.702675    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.718310 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.718282    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"]
Oct 29 22:54:23.729761 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.729734    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:23.729898 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.729818    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:54:23.792393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.792360    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7262786-2553-4cd7-b798-143394491725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.792554 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.792511    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7262786-2553-4cd7-b798-143394491725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.792623 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.792602    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9q2v\" (UniqueName: \"kubernetes.io/projected/f7262786-2553-4cd7-b798-143394491725-kube-api-access-d9q2v\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.893825 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.893770    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9q2v\" (UniqueName: \"kubernetes.io/projected/f7262786-2553-4cd7-b798-143394491725-kube-api-access-d9q2v\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.893825 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.893829    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7262786-2553-4cd7-b798-143394491725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.894044 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.893930    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7262786-2553-4cd7-b798-143394491725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.894551 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.894523    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7262786-2553-4cd7-b798-143394491725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.896456 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.896431    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7262786-2553-4cd7-b798-143394491725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:23.915422 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:23.915390    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9q2v\" (UniqueName: \"kubernetes.io/projected/f7262786-2553-4cd7-b798-143394491725-kube-api-access-d9q2v\") pod \"maas-api-revoke-sub-a294ddb4-4n7vv\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:24.016055 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:24.016023    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:24.160812 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:24.160779    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:24.321002 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:24.320962    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7262786_2553_4cd7_b798_143394491725.slice/crio-82d73aa5af9de544f1acfdfe4f954d9c2ad0173a86089b2367efaf99a75ec4ef WatchSource:0}: Error finding container 82d73aa5af9de544f1acfdfe4f954d9c2ad0173a86089b2367efaf99a75ec4ef: Status 404 returned error can't find the container with id 82d73aa5af9de544f1acfdfe4f954d9c2ad0173a86089b2367efaf99a75ec4ef
Oct 29 22:54:24.321359 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:24.321336    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"]
Oct 29 22:54:24.576177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:24.576118    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv" event={"ID":"f7262786-2553-4cd7-b798-143394491725","Type":"ContainerStarted","Data":"82d73aa5af9de544f1acfdfe4f954d9c2ad0173a86089b2367efaf99a75ec4ef"}
Oct 29 22:54:25.583471 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:25.583435    4839 generic.go:356] "Generic (PLEG): container finished" podID="f7262786-2553-4cd7-b798-143394491725" containerID="735c95c610f00b04a521d208e9ed334823caba1324f8a55003d486d1e41177df" exitCode=0
Oct 29 22:54:25.583901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:25.583528    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv" event={"ID":"f7262786-2553-4cd7-b798-143394491725","Type":"ContainerDied","Data":"735c95c610f00b04a521d208e9ed334823caba1324f8a55003d486d1e41177df"}
Oct 29 22:54:25.977221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:25.977121    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:26.150974 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.150944    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"]
Oct 29 22:54:26.154597 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.154576    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.180815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.175032    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"]
Oct 29 22:54:26.219748 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.219714    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.219906 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.219773    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.219948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.219918    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mb8h\" (UniqueName: \"kubernetes.io/projected/d71c997b-f5e5-4cdd-b015-bed43096e5ce-kube-api-access-7mb8h\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.320720 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.320685    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mb8h\" (UniqueName: \"kubernetes.io/projected/d71c997b-f5e5-4cdd-b015-bed43096e5ce-kube-api-access-7mb8h\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.320877 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.320800    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.320877 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.320844    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.321498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.321472    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.323397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.323370    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.350732 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.350701    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mb8h\" (UniqueName: \"kubernetes.io/projected/d71c997b-f5e5-4cdd-b015-bed43096e5ce-kube-api-access-7mb8h\") pod \"maas-api-revoke-sub-8e6dfd5a-fzjnz\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.468190 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.468131    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:26.726995 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.726969    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:26.764834 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.764807    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"]
Oct 29 22:54:26.764960 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:26.764938    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71c997b_f5e5_4cdd_b015_bed43096e5ce.slice/crio-11c1ea37fdb2c12bff326d1d3399825b5ffea2a29cdea553fb5833f2bb80d2ee WatchSource:0}: Error finding container 11c1ea37fdb2c12bff326d1d3399825b5ffea2a29cdea553fb5833f2bb80d2ee: Status 404 returned error can't find the container with id 11c1ea37fdb2c12bff326d1d3399825b5ffea2a29cdea553fb5833f2bb80d2ee
Oct 29 22:54:26.825268 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.825231    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7262786-2553-4cd7-b798-143394491725-kube-api-access-d9q2v\" (UniqueName: \"kubernetes.io/projected/f7262786-2553-4cd7-b798-143394491725-kube-api-access-d9q2v\") pod \"f7262786-2553-4cd7-b798-143394491725\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") "
Oct 29 22:54:26.825445 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.825316    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7262786-2553-4cd7-b798-143394491725-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7262786-2553-4cd7-b798-143394491725-maas-api-xks-ca\") pod \"f7262786-2553-4cd7-b798-143394491725\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") "
Oct 29 22:54:26.825514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.825463    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7262786-2553-4cd7-b798-143394491725-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7262786-2553-4cd7-b798-143394491725-maas-api-openshift-ca\") pod \"f7262786-2553-4cd7-b798-143394491725\" (UID: \"f7262786-2553-4cd7-b798-143394491725\") "
Oct 29 22:54:26.825812 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.825788    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7262786-2553-4cd7-b798-143394491725-maas-api-openshift-ca" pod "f7262786-2553-4cd7-b798-143394491725" (UID: "f7262786-2553-4cd7-b798-143394491725"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:26.827482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.827459    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7262786-2553-4cd7-b798-143394491725-maas-api-xks-ca" pod "f7262786-2553-4cd7-b798-143394491725" (UID: "f7262786-2553-4cd7-b798-143394491725"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:26.827589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.827545    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7262786-2553-4cd7-b798-143394491725-kube-api-access-d9q2v" pod "f7262786-2553-4cd7-b798-143394491725" (UID: "f7262786-2553-4cd7-b798-143394491725"). InnerVolumeSpecName "kube-api-access-d9q2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:26.927334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.927248    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9q2v\" (UniqueName: \"kubernetes.io/projected/f7262786-2553-4cd7-b798-143394491725-kube-api-access-d9q2v\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.927334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.927277    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7262786-2553-4cd7-b798-143394491725-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:26.927334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:26.927291    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7262786-2553-4cd7-b798-143394491725-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:27.597835 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:27.597735    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"
Oct 29 22:54:27.597835 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:27.597729    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv" event={"ID":"f7262786-2553-4cd7-b798-143394491725","Type":"ContainerDied","Data":"82d73aa5af9de544f1acfdfe4f954d9c2ad0173a86089b2367efaf99a75ec4ef"}
Oct 29 22:54:27.598083 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:27.597899    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d73aa5af9de544f1acfdfe4f954d9c2ad0173a86089b2367efaf99a75ec4ef"
Oct 29 22:54:27.600437 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:27.599993    4839 generic.go:356] "Generic (PLEG): container finished" podID="d71c997b-f5e5-4cdd-b015-bed43096e5ce" containerID="b5b1edd6a5d164323ac48933d64460e8ff8b9c5fd0417205e33c2df8c86ee1ff" exitCode=0
Oct 29 22:54:27.600437 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:27.600070    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz" event={"ID":"d71c997b-f5e5-4cdd-b015-bed43096e5ce","Type":"ContainerDied","Data":"b5b1edd6a5d164323ac48933d64460e8ff8b9c5fd0417205e33c2df8c86ee1ff"}
Oct 29 22:54:27.600437 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:27.600112    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz" event={"ID":"d71c997b-f5e5-4cdd-b015-bed43096e5ce","Type":"ContainerStarted","Data":"11c1ea37fdb2c12bff326d1d3399825b5ffea2a29cdea553fb5833f2bb80d2ee"}
Oct 29 22:54:27.604886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:27.604864    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:28.758001 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.757974    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:28.848084 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.848051    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-xks-ca\") pod \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") "
Oct 29 22:54:28.848286 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.848263    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71c997b-f5e5-4cdd-b015-bed43096e5ce-kube-api-access-7mb8h\" (UniqueName: \"kubernetes.io/projected/d71c997b-f5e5-4cdd-b015-bed43096e5ce-kube-api-access-7mb8h\") pod \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") "
Oct 29 22:54:28.848400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.848384    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-openshift-ca\") pod \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\" (UID: \"d71c997b-f5e5-4cdd-b015-bed43096e5ce\") "
Oct 29 22:54:28.848757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.848732    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-openshift-ca" pod "d71c997b-f5e5-4cdd-b015-bed43096e5ce" (UID: "d71c997b-f5e5-4cdd-b015-bed43096e5ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:28.850255 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.850226    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-xks-ca" pod "d71c997b-f5e5-4cdd-b015-bed43096e5ce" (UID: "d71c997b-f5e5-4cdd-b015-bed43096e5ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:28.850398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.850378    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71c997b-f5e5-4cdd-b015-bed43096e5ce-kube-api-access-7mb8h" pod "d71c997b-f5e5-4cdd-b015-bed43096e5ce" (UID: "d71c997b-f5e5-4cdd-b015-bed43096e5ce"). InnerVolumeSpecName "kube-api-access-7mb8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:28.950267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.950194    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:28.950267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.950218    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d71c997b-f5e5-4cdd-b015-bed43096e5ce-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:28.950267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:28.950227    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mb8h\" (UniqueName: \"kubernetes.io/projected/d71c997b-f5e5-4cdd-b015-bed43096e5ce-kube-api-access-7mb8h\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:29.190864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:29.190835    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:29.617742 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:29.617710    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"
Oct 29 22:54:29.617916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:29.617737    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz" event={"ID":"d71c997b-f5e5-4cdd-b015-bed43096e5ce","Type":"ContainerDied","Data":"11c1ea37fdb2c12bff326d1d3399825b5ffea2a29cdea553fb5833f2bb80d2ee"}
Oct 29 22:54:29.617916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:29.617787    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c1ea37fdb2c12bff326d1d3399825b5ffea2a29cdea553fb5833f2bb80d2ee"
Oct 29 22:54:31.199172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:31.186816    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:32.748878 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:32.748842    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:33.238292 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:54:33.238255    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:54:34.449428 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:34.449393    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:35.929975 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:35.929935    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:37.466954 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:37.466918    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:38.491018 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.490982    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"]
Oct 29 22:54:38.491464 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.491448    4839 cpu_manager.go:414] "containerMap: removing container" podUID="f7262786-2553-4cd7-b798-143394491725" containerName="revoke-keys"
Oct 29 22:54:38.491509 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.491464    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7262786-2553-4cd7-b798-143394491725" containerName="revoke-keys"
Oct 29 22:54:38.491509 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.491493    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d71c997b-f5e5-4cdd-b015-bed43096e5ce" containerName="revoke-keys"
Oct 29 22:54:38.491509 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.491499    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71c997b-f5e5-4cdd-b015-bed43096e5ce" containerName="revoke-keys"
Oct 29 22:54:38.491604 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.491562    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71c997b-f5e5-4cdd-b015-bed43096e5ce" containerName="revoke-keys"
Oct 29 22:54:38.491604 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.491574    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7262786-2553-4cd7-b798-143394491725" containerName="revoke-keys"
Oct 29 22:54:38.495035 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.495017    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.512436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.512402    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"]
Oct 29 22:54:38.517204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.517178    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:38.523540 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.523517    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:54:38.580592 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.580550    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.580737 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.580612    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.580737 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.580705    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chs56\" (UniqueName: \"kubernetes.io/projected/c934a0b9-6681-4d93-830d-ba7fb7334a45-kube-api-access-chs56\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.682035 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.681999    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.682238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.682048    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chs56\" (UniqueName: \"kubernetes.io/projected/c934a0b9-6681-4d93-830d-ba7fb7334a45-kube-api-access-chs56\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.682238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.682138    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.682796 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.682775    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.684446 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.684421    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.703333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.703306    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chs56\" (UniqueName: \"kubernetes.io/projected/c934a0b9-6681-4d93-830d-ba7fb7334a45-kube-api-access-chs56\") pod \"maas-api-revoke-sub-768356ed-bcvbv\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:38.807472 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:38.807435    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:39.104947 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:39.104913    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc934a0b9_6681_4d93_830d_ba7fb7334a45.slice/crio-f6fc4d4560ed8751aff1dc67d1eb27e5177d1e7e224d9c6389b0e101a5196e7c WatchSource:0}: Error finding container f6fc4d4560ed8751aff1dc67d1eb27e5177d1e7e224d9c6389b0e101a5196e7c: Status 404 returned error can't find the container with id f6fc4d4560ed8751aff1dc67d1eb27e5177d1e7e224d9c6389b0e101a5196e7c
Oct 29 22:54:39.105497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:39.105474    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"]
Oct 29 22:54:39.276323 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:39.276294    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:39.678853 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:39.678769    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv" event={"ID":"c934a0b9-6681-4d93-830d-ba7fb7334a45","Type":"ContainerStarted","Data":"1648c059da92e742ff7fb2cd154f76b0e9579d0c56ac0989a88782044fa8c1dc"}
Oct 29 22:54:39.678853 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:39.678829    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv" event={"ID":"c934a0b9-6681-4d93-830d-ba7fb7334a45","Type":"ContainerStarted","Data":"f6fc4d4560ed8751aff1dc67d1eb27e5177d1e7e224d9c6389b0e101a5196e7c"}
Oct 29 22:54:39.731764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:39.731711    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv" podStartSLOduration=1.731697821 podStartE2EDuration="1.731697821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:39.731387651 +0000 UTC m=+736.834463923" watchObservedRunningTime="2026-10-29 22:54:39.731697821 +0000 UTC m=+736.834774091"
Oct 29 22:54:40.686544 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:40.686507    4839 generic.go:356] "Generic (PLEG): container finished" podID="c934a0b9-6681-4d93-830d-ba7fb7334a45" containerID="1648c059da92e742ff7fb2cd154f76b0e9579d0c56ac0989a88782044fa8c1dc" exitCode=0
Oct 29 22:54:40.686912 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:40.686597    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv" event={"ID":"c934a0b9-6681-4d93-830d-ba7fb7334a45","Type":"ContainerDied","Data":"1648c059da92e742ff7fb2cd154f76b0e9579d0c56ac0989a88782044fa8c1dc"}
Oct 29 22:54:40.976440 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:40.976352    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:41.846726 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:41.846701    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:41.911597 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:41.911566    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c934a0b9-6681-4d93-830d-ba7fb7334a45-kube-api-access-chs56\" (UniqueName: \"kubernetes.io/projected/c934a0b9-6681-4d93-830d-ba7fb7334a45-kube-api-access-chs56\") pod \"c934a0b9-6681-4d93-830d-ba7fb7334a45\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") "
Oct 29 22:54:41.911755 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:41.911604    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-xks-ca\") pod \"c934a0b9-6681-4d93-830d-ba7fb7334a45\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") "
Oct 29 22:54:41.911755 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:41.911683    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-openshift-ca\") pod \"c934a0b9-6681-4d93-830d-ba7fb7334a45\" (UID: \"c934a0b9-6681-4d93-830d-ba7fb7334a45\") "
Oct 29 22:54:41.912125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:41.912099    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-openshift-ca" pod "c934a0b9-6681-4d93-830d-ba7fb7334a45" (UID: "c934a0b9-6681-4d93-830d-ba7fb7334a45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:41.913808 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:41.913776    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-xks-ca" pod "c934a0b9-6681-4d93-830d-ba7fb7334a45" (UID: "c934a0b9-6681-4d93-830d-ba7fb7334a45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:41.913918 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:41.913828    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c934a0b9-6681-4d93-830d-ba7fb7334a45-kube-api-access-chs56" pod "c934a0b9-6681-4d93-830d-ba7fb7334a45" (UID: "c934a0b9-6681-4d93-830d-ba7fb7334a45"). InnerVolumeSpecName "kube-api-access-chs56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:42.013054 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.012974    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:42.013054 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.013007    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chs56\" (UniqueName: \"kubernetes.io/projected/c934a0b9-6681-4d93-830d-ba7fb7334a45-kube-api-access-chs56\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:42.013054 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.013018    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c934a0b9-6681-4d93-830d-ba7fb7334a45-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:42.016731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.016703    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"]
Oct 29 22:54:42.017156 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.017132    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c934a0b9-6681-4d93-830d-ba7fb7334a45" containerName="revoke-keys"
Oct 29 22:54:42.017219 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.017175    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c934a0b9-6681-4d93-830d-ba7fb7334a45" containerName="revoke-keys"
Oct 29 22:54:42.017296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.017277    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c934a0b9-6681-4d93-830d-ba7fb7334a45" containerName="revoke-keys"
Oct 29 22:54:42.020528 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.020509    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.039619 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.039590    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"]
Oct 29 22:54:42.113516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.113484    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md8tr\" (UniqueName: \"kubernetes.io/projected/3763dc23-27ce-409e-b286-8e98c1083ee2-kube-api-access-md8tr\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.113714 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.113578    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.113714 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.113671    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.133976 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.133942    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"]
Oct 29 22:54:42.137757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.137735    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.155412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.155385    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"]
Oct 29 22:54:42.214952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.214919    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8q28\" (UniqueName: \"kubernetes.io/projected/8214a394-2030-4c68-ac32-eba04c461ca2-kube-api-access-s8q28\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.214952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.214962    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.215196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.215082    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md8tr\" (UniqueName: \"kubernetes.io/projected/3763dc23-27ce-409e-b286-8e98c1083ee2-kube-api-access-md8tr\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.215196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.215123    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.215196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.215185    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.215309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.215208    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.215861 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.215834    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.217606 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.217581    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.236690 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.236667    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md8tr\" (UniqueName: \"kubernetes.io/projected/3763dc23-27ce-409e-b286-8e98c1083ee2-kube-api-access-md8tr\") pod \"maas-api-revoke-sub-ec018b88-ldmg9\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.316120 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.316088    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.316329 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.316225    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8q28\" (UniqueName: \"kubernetes.io/projected/8214a394-2030-4c68-ac32-eba04c461ca2-kube-api-access-s8q28\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.316329 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.316253    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.316886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.316855    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.318587 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.318568    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.334021 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.333998    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:42.336979 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.336957    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8q28\" (UniqueName: \"kubernetes.io/projected/8214a394-2030-4c68-ac32-eba04c461ca2-kube-api-access-s8q28\") pod \"maas-api-revoke-sub-c712ec65-lwnlb\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.451122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.451089    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:42.630332 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:42.630300    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3763dc23_27ce_409e_b286_8e98c1083ee2.slice/crio-e71cd62cf1f8fea3da22a63b03bdf52a7b6f9a7a091c1a0f32a77d6c93a6d505 WatchSource:0}: Error finding container e71cd62cf1f8fea3da22a63b03bdf52a7b6f9a7a091c1a0f32a77d6c93a6d505: Status 404 returned error can't find the container with id e71cd62cf1f8fea3da22a63b03bdf52a7b6f9a7a091c1a0f32a77d6c93a6d505
Oct 29 22:54:42.630585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.630562    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"]
Oct 29 22:54:42.699578 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.699542    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"
Oct 29 22:54:42.699757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.699543    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv" event={"ID":"c934a0b9-6681-4d93-830d-ba7fb7334a45","Type":"ContainerDied","Data":"f6fc4d4560ed8751aff1dc67d1eb27e5177d1e7e224d9c6389b0e101a5196e7c"}
Oct 29 22:54:42.699757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.699731    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6fc4d4560ed8751aff1dc67d1eb27e5177d1e7e224d9c6389b0e101a5196e7c"
Oct 29 22:54:42.701403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.701371    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9" event={"ID":"3763dc23-27ce-409e-b286-8e98c1083ee2","Type":"ContainerStarted","Data":"e71cd62cf1f8fea3da22a63b03bdf52a7b6f9a7a091c1a0f32a77d6c93a6d505"}
Oct 29 22:54:42.744513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.744460    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:42.745910 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:42.745887    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8214a394_2030_4c68_ac32_eba04c461ca2.slice/crio-2a33ddb858a6975e20eb687099d3f38381187d5cc1b04c0f4ae6127b34a38fa5 WatchSource:0}: Error finding container 2a33ddb858a6975e20eb687099d3f38381187d5cc1b04c0f4ae6127b34a38fa5: Status 404 returned error can't find the container with id 2a33ddb858a6975e20eb687099d3f38381187d5cc1b04c0f4ae6127b34a38fa5
Oct 29 22:54:42.746777 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:42.746748    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"]
Oct 29 22:54:43.713659 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:43.713617    4839 generic.go:356] "Generic (PLEG): container finished" podID="8214a394-2030-4c68-ac32-eba04c461ca2" containerID="046d01dc7f8f05905e550910e254440f45ece7a6d8b3f255392a997ac53bd1c0" exitCode=0
Oct 29 22:54:43.714101 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:43.713779    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb" event={"ID":"8214a394-2030-4c68-ac32-eba04c461ca2","Type":"ContainerDied","Data":"046d01dc7f8f05905e550910e254440f45ece7a6d8b3f255392a997ac53bd1c0"}
Oct 29 22:54:43.714101 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:43.713864    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb" event={"ID":"8214a394-2030-4c68-ac32-eba04c461ca2","Type":"ContainerStarted","Data":"2a33ddb858a6975e20eb687099d3f38381187d5cc1b04c0f4ae6127b34a38fa5"}
Oct 29 22:54:43.720414 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:43.720387    4839 generic.go:356] "Generic (PLEG): container finished" podID="3763dc23-27ce-409e-b286-8e98c1083ee2" containerID="baff145962dab3e7c234b6b16dbfd2d0942496c6a3b38a6452d9a5d90a30c5ba" exitCode=0
Oct 29 22:54:43.720533 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:43.720457    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9" event={"ID":"3763dc23-27ce-409e-b286-8e98c1083ee2","Type":"ContainerDied","Data":"baff145962dab3e7c234b6b16dbfd2d0942496c6a3b38a6452d9a5d90a30c5ba"}
Oct 29 22:54:44.403651 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:44.403604    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:44.977917 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:44.977579    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:45.038320 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.038287    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:45.063888 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.063850    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-xks-ca\") pod \"8214a394-2030-4c68-ac32-eba04c461ca2\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") "
Oct 29 22:54:45.064087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.063916    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-openshift-ca\") pod \"3763dc23-27ce-409e-b286-8e98c1083ee2\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") "
Oct 29 22:54:45.064087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.063956    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3763dc23-27ce-409e-b286-8e98c1083ee2-kube-api-access-md8tr\" (UniqueName: \"kubernetes.io/projected/3763dc23-27ce-409e-b286-8e98c1083ee2-kube-api-access-md8tr\") pod \"3763dc23-27ce-409e-b286-8e98c1083ee2\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") "
Oct 29 22:54:45.064087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.063988    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8214a394-2030-4c68-ac32-eba04c461ca2-kube-api-access-s8q28\" (UniqueName: \"kubernetes.io/projected/8214a394-2030-4c68-ac32-eba04c461ca2-kube-api-access-s8q28\") pod \"8214a394-2030-4c68-ac32-eba04c461ca2\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") "
Oct 29 22:54:45.064087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.064070    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-openshift-ca\") pod \"8214a394-2030-4c68-ac32-eba04c461ca2\" (UID: \"8214a394-2030-4c68-ac32-eba04c461ca2\") "
Oct 29 22:54:45.064369 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.064205    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-xks-ca\") pod \"3763dc23-27ce-409e-b286-8e98c1083ee2\" (UID: \"3763dc23-27ce-409e-b286-8e98c1083ee2\") "
Oct 29 22:54:45.065069 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.064724    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-openshift-ca" pod "3763dc23-27ce-409e-b286-8e98c1083ee2" (UID: "3763dc23-27ce-409e-b286-8e98c1083ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:45.065852 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.065754    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-openshift-ca" pod "8214a394-2030-4c68-ac32-eba04c461ca2" (UID: "8214a394-2030-4c68-ac32-eba04c461ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:45.075195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.067115    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-xks-ca" pod "8214a394-2030-4c68-ac32-eba04c461ca2" (UID: "8214a394-2030-4c68-ac32-eba04c461ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:45.075195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.067271    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-xks-ca" pod "3763dc23-27ce-409e-b286-8e98c1083ee2" (UID: "3763dc23-27ce-409e-b286-8e98c1083ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:45.075195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.068320    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3763dc23-27ce-409e-b286-8e98c1083ee2-kube-api-access-md8tr" pod "3763dc23-27ce-409e-b286-8e98c1083ee2" (UID: "3763dc23-27ce-409e-b286-8e98c1083ee2"). InnerVolumeSpecName "kube-api-access-md8tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:45.081005 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.080936    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8214a394-2030-4c68-ac32-eba04c461ca2-kube-api-access-s8q28" pod "8214a394-2030-4c68-ac32-eba04c461ca2" (UID: "8214a394-2030-4c68-ac32-eba04c461ca2"). InnerVolumeSpecName "kube-api-access-s8q28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:45.165928 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.165891    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.165928 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.165927    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.166184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.165940    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8214a394-2030-4c68-ac32-eba04c461ca2-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.166184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.165952    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3763dc23-27ce-409e-b286-8e98c1083ee2-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.166184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.165964    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md8tr\" (UniqueName: \"kubernetes.io/projected/3763dc23-27ce-409e-b286-8e98c1083ee2-kube-api-access-md8tr\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.166184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.165978    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8q28\" (UniqueName: \"kubernetes.io/projected/8214a394-2030-4c68-ac32-eba04c461ca2-kube-api-access-s8q28\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:45.746194 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.746120    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb" event={"ID":"8214a394-2030-4c68-ac32-eba04c461ca2","Type":"ContainerDied","Data":"2a33ddb858a6975e20eb687099d3f38381187d5cc1b04c0f4ae6127b34a38fa5"}
Oct 29 22:54:45.746373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.746220    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a33ddb858a6975e20eb687099d3f38381187d5cc1b04c0f4ae6127b34a38fa5"
Oct 29 22:54:45.746373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.746312    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"
Oct 29 22:54:45.750351 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.750328    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"
Oct 29 22:54:45.750482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.750405    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9" event={"ID":"3763dc23-27ce-409e-b286-8e98c1083ee2","Type":"ContainerDied","Data":"e71cd62cf1f8fea3da22a63b03bdf52a7b6f9a7a091c1a0f32a77d6c93a6d505"}
Oct 29 22:54:45.751217 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:45.751194    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e71cd62cf1f8fea3da22a63b03bdf52a7b6f9a7a091c1a0f32a77d6c93a6d505"
Oct 29 22:54:46.341172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:46.340723    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:48.053133 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:48.053099    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:49.984489 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:49.984453    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:51.509931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.509894    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:51.819063 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.819031    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"]
Oct 29 22:54:51.819541 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.819525    4839 cpu_manager.go:414] "containerMap: removing container" podUID="3763dc23-27ce-409e-b286-8e98c1083ee2" containerName="revoke-keys"
Oct 29 22:54:51.819541 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.819541    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3763dc23-27ce-409e-b286-8e98c1083ee2" containerName="revoke-keys"
Oct 29 22:54:51.819677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.819558    4839 cpu_manager.go:414] "containerMap: removing container" podUID="8214a394-2030-4c68-ac32-eba04c461ca2" containerName="revoke-keys"
Oct 29 22:54:51.819677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.819563    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8214a394-2030-4c68-ac32-eba04c461ca2" containerName="revoke-keys"
Oct 29 22:54:51.819677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.819646    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="3763dc23-27ce-409e-b286-8e98c1083ee2" containerName="revoke-keys"
Oct 29 22:54:51.819677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.819659    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="8214a394-2030-4c68-ac32-eba04c461ca2" containerName="revoke-keys"
Oct 29 22:54:51.824998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.824978    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:51.839980 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.839949    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"]
Oct 29 22:54:51.850681 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.850655    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:54:51.850838 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.850747    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:54:51.926806 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.926772    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:51.926968 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.926830    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:51.926968 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:51.926961    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthrf\" (UniqueName: \"kubernetes.io/projected/46c78859-6b20-426d-91e1-4d968380fe6d-kube-api-access-jthrf\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.027903 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.027862    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthrf\" (UniqueName: \"kubernetes.io/projected/46c78859-6b20-426d-91e1-4d968380fe6d-kube-api-access-jthrf\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.028082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.027947    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.028082 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.027991    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.028667 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.028634    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.030380 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.030354    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.049515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.049489    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthrf\" (UniqueName: \"kubernetes.io/projected/46c78859-6b20-426d-91e1-4d968380fe6d-kube-api-access-jthrf\") pod \"maas-api-revoke-sub-d4ab029a-lz5gb\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.139179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.139053    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:52.339737 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.339705    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"]
Oct 29 22:54:52.345208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.345187    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.374669 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.368363    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"]
Oct 29 22:54:52.434302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.434203    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.434302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.434264    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.434437 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.434408    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dnnk\" (UniqueName: \"kubernetes.io/projected/c104c251-13d8-4bee-9888-5f3c381c1d29-kube-api-access-6dnnk\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.436770 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:52.436738    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c78859_6b20_426d_91e1_4d968380fe6d.slice/crio-e454a7e23afef7d27e5a15be53fbb514ac2a591a4a44a478561b05cfecc33519 WatchSource:0}: Error finding container e454a7e23afef7d27e5a15be53fbb514ac2a591a4a44a478561b05cfecc33519: Status 404 returned error can't find the container with id e454a7e23afef7d27e5a15be53fbb514ac2a591a4a44a478561b05cfecc33519
Oct 29 22:54:52.437049 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.437026    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"]
Oct 29 22:54:52.464981 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.464955    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"]
Oct 29 22:54:52.470491 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.470468    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.486763 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.486734    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"]
Oct 29 22:54:52.490943 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.490916    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"kube-root-ca.crt\""
Oct 29 22:54:52.491049 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.490940    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"openshift-service-ca.crt\""
Oct 29 22:54:52.491170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.491130    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"default-dockercfg-spmz9\""
Oct 29 22:54:52.491307 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.491290    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"distinct-w4-dd3ff3-kserve-self-signed-certs\""
Oct 29 22:54:52.526912 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.526884    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:52.535242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.535213    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.535382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.535255    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.535382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.535299    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dnnk\" (UniqueName: \"kubernetes.io/projected/c104c251-13d8-4bee-9888-5f3c381c1d29-kube-api-access-6dnnk\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.536040 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.536015    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.537939 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.537917    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.556699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.556664    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dnnk\" (UniqueName: \"kubernetes.io/projected/c104c251-13d8-4bee-9888-5f3c381c1d29-kube-api-access-6dnnk\") pod \"maas-api-revoke-sub-e5ea6316-p7rzg\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.636615 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.636585    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-model-cache\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.636764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.636643    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-dshm\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.636764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.636735    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8xw4\" (UniqueName: \"kubernetes.io/projected/ecc56888-0e26-441a-a3d4-da19bba238ee-kube-api-access-v8xw4\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.636847 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.636768    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-tmp-dir\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.636847 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.636836    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc56888-0e26-441a-a3d4-da19bba238ee-tls-certs\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.636908 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.636883    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-home\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.688536 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.685819    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:52.738901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.738230    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-dshm\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.738901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.738287    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8xw4\" (UniqueName: \"kubernetes.io/projected/ecc56888-0e26-441a-a3d4-da19bba238ee-kube-api-access-v8xw4\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.738901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.738329    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-tmp-dir\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.738901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.738397    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc56888-0e26-441a-a3d4-da19bba238ee-tls-certs\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.738901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.738425    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-home\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.738901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.738517    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-model-cache\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.739341 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.738981    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-model-cache\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.739341 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.739138    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-home\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.739341 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.739182    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-tmp-dir\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.740636 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.740609    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-dshm\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.741252 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.741231    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc56888-0e26-441a-a3d4-da19bba238ee-tls-certs\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.759574 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.759549    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8xw4\" (UniqueName: \"kubernetes.io/projected/ecc56888-0e26-441a-a3d4-da19bba238ee-kube-api-access-v8xw4\") pod \"distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.808688 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.807511    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:52.819595 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.819555    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb" event={"ID":"46c78859-6b20-426d-91e1-4d968380fe6d","Type":"ContainerStarted","Data":"c8bcd49a7d7ddc15ef344e2a3707f53b912a9187789d1f863f02a0bcbef994db"}
Oct 29 22:54:52.819753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:52.819625    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb" event={"ID":"46c78859-6b20-426d-91e1-4d968380fe6d","Type":"ContainerStarted","Data":"e454a7e23afef7d27e5a15be53fbb514ac2a591a4a44a478561b05cfecc33519"}
Oct 29 22:54:53.028320 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.028265    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb" podStartSLOduration=2.028249811 podStartE2EDuration="2.028249811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:52.863588366 +0000 UTC m=+749.966664637" watchObservedRunningTime="2026-10-29 22:54:53.028249811 +0000 UTC m=+750.131326083"
Oct 29 22:54:53.028651 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:53.028619    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc104c251_13d8_4bee_9888_5f3c381c1d29.slice/crio-35b7ce546baf2a6f9169f8c8a53998a1ad59f35862c607ba08a5fbe239408820 WatchSource:0}: Error finding container 35b7ce546baf2a6f9169f8c8a53998a1ad59f35862c607ba08a5fbe239408820: Status 404 returned error can't find the container with id 35b7ce546baf2a6f9169f8c8a53998a1ad59f35862c607ba08a5fbe239408820
Oct 29 22:54:53.029268 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.029129    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"]
Oct 29 22:54:53.114695 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:53.114595    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecc56888_0e26_441a_a3d4_da19bba238ee.slice/crio-c46a74e518148d540aafc7bf311aa94f399d2aafdb25e772f194544b25e77d35 WatchSource:0}: Error finding container c46a74e518148d540aafc7bf311aa94f399d2aafdb25e772f194544b25e77d35: Status 404 returned error can't find the container with id c46a74e518148d540aafc7bf311aa94f399d2aafdb25e772f194544b25e77d35
Oct 29 22:54:53.115137 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.115113    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"]
Oct 29 22:54:53.160870 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.160840    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:53.827081 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.827046    4839 generic.go:356] "Generic (PLEG): container finished" podID="46c78859-6b20-426d-91e1-4d968380fe6d" containerID="c8bcd49a7d7ddc15ef344e2a3707f53b912a9187789d1f863f02a0bcbef994db" exitCode=0
Oct 29 22:54:53.827508 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.827130    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb" event={"ID":"46c78859-6b20-426d-91e1-4d968380fe6d","Type":"ContainerDied","Data":"c8bcd49a7d7ddc15ef344e2a3707f53b912a9187789d1f863f02a0bcbef994db"}
Oct 29 22:54:53.828733 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.828710    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg" event={"ID":"c104c251-13d8-4bee-9888-5f3c381c1d29","Type":"ContainerStarted","Data":"7ef9288b9e0afc68058a7bb3fde6c4f8fd2660161c82b2493cd41b4d863b78da"}
Oct 29 22:54:53.828878 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.828756    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg" event={"ID":"c104c251-13d8-4bee-9888-5f3c381c1d29","Type":"ContainerStarted","Data":"35b7ce546baf2a6f9169f8c8a53998a1ad59f35862c607ba08a5fbe239408820"}
Oct 29 22:54:53.830295 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.830276    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" event={"ID":"ecc56888-0e26-441a-a3d4-da19bba238ee","Type":"ContainerStarted","Data":"aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572"}
Oct 29 22:54:53.830383 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.830301    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" event={"ID":"ecc56888-0e26-441a-a3d4-da19bba238ee","Type":"ContainerStarted","Data":"c46a74e518148d540aafc7bf311aa94f399d2aafdb25e772f194544b25e77d35"}
Oct 29 22:54:53.911970 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.911914    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg" podStartSLOduration=1.9119019430000002 podStartE2EDuration="1.911901943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:53.910208949 +0000 UTC m=+751.013285221" watchObservedRunningTime="2026-10-29 22:54:53.911901943 +0000 UTC m=+751.014978214"
Oct 29 22:54:53.953066 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:53.953017    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podStartSLOduration=1.95300395 podStartE2EDuration="1.95300395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:53.952069195 +0000 UTC m=+751.055145668" watchObservedRunningTime="2026-10-29 22:54:53.95300395 +0000 UTC m=+751.056080221"
Oct 29 22:54:54.808711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:54.808674    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:54:54.844856 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:54.844820    4839 generic.go:356] "Generic (PLEG): container finished" podID="c104c251-13d8-4bee-9888-5f3c381c1d29" containerID="7ef9288b9e0afc68058a7bb3fde6c4f8fd2660161c82b2493cd41b4d863b78da" exitCode=0
Oct 29 22:54:54.845370 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:54.844900    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg" event={"ID":"c104c251-13d8-4bee-9888-5f3c381c1d29","Type":"ContainerDied","Data":"7ef9288b9e0afc68058a7bb3fde6c4f8fd2660161c82b2493cd41b4d863b78da"}
Oct 29 22:54:54.875616 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:54.874355    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:55.021442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.021413    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:55.064051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.063951    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-openshift-ca\") pod \"46c78859-6b20-426d-91e1-4d968380fe6d\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") "
Oct 29 22:54:55.064051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.064013    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c78859-6b20-426d-91e1-4d968380fe6d-kube-api-access-jthrf\" (UniqueName: \"kubernetes.io/projected/46c78859-6b20-426d-91e1-4d968380fe6d-kube-api-access-jthrf\") pod \"46c78859-6b20-426d-91e1-4d968380fe6d\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") "
Oct 29 22:54:55.064313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.064284    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-xks-ca\") pod \"46c78859-6b20-426d-91e1-4d968380fe6d\" (UID: \"46c78859-6b20-426d-91e1-4d968380fe6d\") "
Oct 29 22:54:55.064436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.064402    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-openshift-ca" pod "46c78859-6b20-426d-91e1-4d968380fe6d" (UID: "46c78859-6b20-426d-91e1-4d968380fe6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:55.064764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.064741    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:55.067276 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.067250    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c78859-6b20-426d-91e1-4d968380fe6d-kube-api-access-jthrf" pod "46c78859-6b20-426d-91e1-4d968380fe6d" (UID: "46c78859-6b20-426d-91e1-4d968380fe6d"). InnerVolumeSpecName "kube-api-access-jthrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:55.068777 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.068750    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-xks-ca" pod "46c78859-6b20-426d-91e1-4d968380fe6d" (UID: "46c78859-6b20-426d-91e1-4d968380fe6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:55.165936 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.165903    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jthrf\" (UniqueName: \"kubernetes.io/projected/46c78859-6b20-426d-91e1-4d968380fe6d-kube-api-access-jthrf\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:55.165936 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.165934    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46c78859-6b20-426d-91e1-4d968380fe6d-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:55.779774 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.779735    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"]
Oct 29 22:54:55.780477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.780456    4839 cpu_manager.go:414] "containerMap: removing container" podUID="46c78859-6b20-426d-91e1-4d968380fe6d" containerName="revoke-keys"
Oct 29 22:54:55.780547 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.780481    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c78859-6b20-426d-91e1-4d968380fe6d" containerName="revoke-keys"
Oct 29 22:54:55.780637 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.780625    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c78859-6b20-426d-91e1-4d968380fe6d" containerName="revoke-keys"
Oct 29 22:54:55.784569 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.784529    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.821954 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.821922    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"]
Oct 29 22:54:55.853086 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.853056    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"
Oct 29 22:54:55.853086 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.853077    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb" event={"ID":"46c78859-6b20-426d-91e1-4d968380fe6d","Type":"ContainerDied","Data":"e454a7e23afef7d27e5a15be53fbb514ac2a591a4a44a478561b05cfecc33519"}
Oct 29 22:54:55.853598 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.853126    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e454a7e23afef7d27e5a15be53fbb514ac2a591a4a44a478561b05cfecc33519"
Oct 29 22:54:55.873020 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.872988    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gvq\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-kube-api-access-49gvq\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873029    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873057    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873274    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873445 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873417    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30d17cf-246c-4524-a1b2-78d4c34fc793-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873543 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873525    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873593 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873567    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873686 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873669    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.873746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.873717    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.883516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.883492    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ffba57-openshift-default-dockercfg-5l7zt\""
Oct 29 22:54:55.938035 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.938000    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"]
Oct 29 22:54:55.943041 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.943017    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.963400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.963372    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"]
Oct 29 22:54:55.974766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.974739    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.974898 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.974795    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.974898 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.974838    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.974982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.974910    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gvq\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-kube-api-access-49gvq\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.974982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.974932    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.974982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.974953    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.974982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.974978    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463fcbe-ed3c-4234-828d-955fa96a394a-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.975226 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975011    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.975226 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975095    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.975226 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975206    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30d17cf-246c-4524-a1b2-78d4c34fc793-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.975398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975253    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.975398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975290    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.975398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975328    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.975398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975375    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.975800 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975408    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgk5\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-kube-api-access-shgk5\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.975800 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975448    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.975800 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975484    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.975800 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975557    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:55.975800 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.975562    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.976170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.976015    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.976170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.976069    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.976766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.976495    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30d17cf-246c-4524-a1b2-78d4c34fc793-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.976946 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.976921    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.977966 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.977937    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.979363 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.979343    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.996134 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.996108    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gvq\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-kube-api-access-49gvq\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:55.996531 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:55.996509    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:56.000443 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.000421    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:56.076781 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.076692    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-openshift-ca\") pod \"c104c251-13d8-4bee-9888-5f3c381c1d29\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") "
Oct 29 22:54:56.076947 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.076786    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-xks-ca\") pod \"c104c251-13d8-4bee-9888-5f3c381c1d29\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") "
Oct 29 22:54:56.076947 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.076826    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c104c251-13d8-4bee-9888-5f3c381c1d29-kube-api-access-6dnnk\" (UniqueName: \"kubernetes.io/projected/c104c251-13d8-4bee-9888-5f3c381c1d29-kube-api-access-6dnnk\") pod \"c104c251-13d8-4bee-9888-5f3c381c1d29\" (UID: \"c104c251-13d8-4bee-9888-5f3c381c1d29\") "
Oct 29 22:54:56.077211 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077168    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-openshift-ca" pod "c104c251-13d8-4bee-9888-5f3c381c1d29" (UID: "c104c251-13d8-4bee-9888-5f3c381c1d29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:54:56.077561 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077521    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.077655 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077617    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.077655 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077642    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.077766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077702    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.077766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077741    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463fcbe-ed3c-4234-828d-955fa96a394a-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.077873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077858    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.077990 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.077912    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgk5\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-kube-api-access-shgk5\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.078115 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.078023    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.078115 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.078063    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.078279 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.078139    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:56.078529 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.078507    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.078864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.078764    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.078864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.078807    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463fcbe-ed3c-4234-828d-955fa96a394a-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.079350 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.079287    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.079616 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.079583    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-xks-ca" pod "c104c251-13d8-4bee-9888-5f3c381c1d29" (UID: "c104c251-13d8-4bee-9888-5f3c381c1d29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:54:56.079723 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.079690    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c104c251-13d8-4bee-9888-5f3c381c1d29-kube-api-access-6dnnk" pod "c104c251-13d8-4bee-9888-5f3c381c1d29" (UID: "c104c251-13d8-4bee-9888-5f3c381c1d29"). InnerVolumeSpecName "kube-api-access-6dnnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:54:56.079896 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.079876    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.081417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.081390    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.082186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.082135    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.099905 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.099874    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.100015 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.099950    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgk5\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-kube-api-access-shgk5\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.100729 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.100708    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:56.179484 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.179444    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c104c251-13d8-4bee-9888-5f3c381c1d29-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:56.179484 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.179488    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dnnk\" (UniqueName: \"kubernetes.io/projected/c104c251-13d8-4bee-9888-5f3c381c1d29-kube-api-access-6dnnk\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:54:56.265535 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.265497    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:56.422194 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:56.422163    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30d17cf_246c_4524_a1b2_78d4c34fc793.slice/crio-1d606112affcfb30bb48052e824bb00969dcb31eb45e9b58499e47686d0c02d5 WatchSource:0}: Error finding container 1d606112affcfb30bb48052e824bb00969dcb31eb45e9b58499e47686d0c02d5: Status 404 returned error can't find the container with id 1d606112affcfb30bb48052e824bb00969dcb31eb45e9b58499e47686d0c02d5
Oct 29 22:54:56.423101 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.423076    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"]
Oct 29 22:54:56.424766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.424737    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:56.424840 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.424796    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:56.424840 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.424824    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:56.566864 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:54:56.566819    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7463fcbe_ed3c_4234_828d_955fa96a394a.slice/crio-1adacba6ce777faff85aac9480bc47b06d1e6573d0b49cc2faa1f409d48e6429 WatchSource:0}: Error finding container 1adacba6ce777faff85aac9480bc47b06d1e6573d0b49cc2faa1f409d48e6429: Status 404 returned error can't find the container with id 1adacba6ce777faff85aac9480bc47b06d1e6573d0b49cc2faa1f409d48e6429
Oct 29 22:54:56.568653 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.568627    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"]
Oct 29 22:54:56.569693 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.569653    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:56.569779 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.569732    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:56.569779 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.569773    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:54:56.861660 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.861620    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" event={"ID":"b30d17cf-246c-4524-a1b2-78d4c34fc793","Type":"ContainerStarted","Data":"9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a"}
Oct 29 22:54:56.862171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.861683    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" event={"ID":"b30d17cf-246c-4524-a1b2-78d4c34fc793","Type":"ContainerStarted","Data":"1d606112affcfb30bb48052e824bb00969dcb31eb45e9b58499e47686d0c02d5"}
Oct 29 22:54:56.863674 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.863642    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg" event={"ID":"c104c251-13d8-4bee-9888-5f3c381c1d29","Type":"ContainerDied","Data":"35b7ce546baf2a6f9169f8c8a53998a1ad59f35862c607ba08a5fbe239408820"}
Oct 29 22:54:56.863822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.863697    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b7ce546baf2a6f9169f8c8a53998a1ad59f35862c607ba08a5fbe239408820"
Oct 29 22:54:56.863822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.863756    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"
Oct 29 22:54:56.877046 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.877003    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" event={"ID":"7463fcbe-ed3c-4234-828d-955fa96a394a","Type":"ContainerStarted","Data":"0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d"}
Oct 29 22:54:56.877213 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.877103    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" event={"ID":"7463fcbe-ed3c-4234-828d-955fa96a394a","Type":"ContainerStarted","Data":"1adacba6ce777faff85aac9480bc47b06d1e6573d0b49cc2faa1f409d48e6429"}
Oct 29 22:54:56.907077 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.907022    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podStartSLOduration=1.9070084619999998 podStartE2EDuration="1.907008462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:56.906203997 +0000 UTC m=+754.009280267" watchObservedRunningTime="2026-10-29 22:54:56.907008462 +0000 UTC m=+754.010084733"
Oct 29 22:54:56.969461 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.969401    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podStartSLOduration=1.969380969 podStartE2EDuration="1.969380969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:54:56.967990999 +0000 UTC m=+754.071067271" watchObservedRunningTime="2026-10-29 22:54:56.969380969 +0000 UTC m=+754.072457242"
Oct 29 22:54:56.973334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:56.973286    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:57.101026 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:57.100986    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:54:57.103016 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:57.102986    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:54:57.103135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:57.103033    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:54:57.266370 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:57.266333    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:54:57.267616 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:57.267591    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:54:57.267733 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:57.267638    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:54:58.103609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:58.103570    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:54:58.104204 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:58.104171    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:54:58.266540 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:58.266501    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:54:58.266718 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:58.266592    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:54:58.912107 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:58.912070    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:54:59.102605 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:59.102563    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:54:59.102876 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:59.102843    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:54:59.274715 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:59.273603    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:54:59.274715 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:54:59.273667    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:55:00.106173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:00.106034    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:55:00.106173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:00.106093    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:55:00.268175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:00.266669    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:55:00.268175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:00.266738    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:55:01.100170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.096241    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:01.106176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.101511    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:55:01.106176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.101567    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:55:01.154327 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.153804    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:01.217910 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.217295    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:01.268715 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.268646    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:55:01.269009 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.268955    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:55:01.320292 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.320239    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:01.426228 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.426018    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:01.923307 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.923267    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:01.923636 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:01.923597    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.098300 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.098237    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.099336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.099311    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:55:02.101179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.101137    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:55:02.101695 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.101665    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:55:02.155272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.155227    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.157689 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.157660    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:55:02.310170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.309342    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:55:02.310170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.309424    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:55:02.757223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.749775    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"]
Oct 29 22:55:02.767177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.758101    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c104c251-13d8-4bee-9888-5f3c381c1d29" containerName="revoke-keys"
Oct 29 22:55:02.767427 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.767395    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c104c251-13d8-4bee-9888-5f3c381c1d29" containerName="revoke-keys"
Oct 29 22:55:02.789173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.767722    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c104c251-13d8-4bee-9888-5f3c381c1d29" containerName="revoke-keys"
Oct 29 22:55:02.886223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.862303    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:02.897342 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.897306    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"]
Oct 29 22:55:02.897602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.897584    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:55:02.907171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.898556    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:02.933431 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.929267    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:55:02.943283 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:02.942383    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:55:03.083561 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.074552    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.083561 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.074620    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztwc\" (UniqueName: \"kubernetes.io/projected/b1972eab-70dc-45f5-8f92-3061b0d862c4-kube-api-access-6ztwc\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.083561 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.074658    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.102175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.097924    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rntkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:03.102175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.097986    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rntkv" podUID="80f4fedc-6afb-42e4-87de-8b77868b9125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:03.147174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.143514    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:55:03.147174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.144581    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:55:03.147174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.144625    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:55:03.147174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.145097    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:03.189171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.188692    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.189171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.188759    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztwc\" (UniqueName: \"kubernetes.io/projected/b1972eab-70dc-45f5-8f92-3061b0d862c4-kube-api-access-6ztwc\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.189171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.188799    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.189728 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.189411    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:55:03.199176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.190759    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.215789 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.215747    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-84ct6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:03.224259 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.224214    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-84ct6" podUID="f4957a03-1161-47a8-a4c1-69c75f901597" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:03.227671 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.227638    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.274172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.251363    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:03.274172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.273439    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:55:03.274172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.273506    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:55:03.288172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.280880    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztwc\" (UniqueName: \"kubernetes.io/projected/b1972eab-70dc-45f5-8f92-3061b0d862c4-kube-api-access-6ztwc\") pod \"maas-api-revoke-sub-e70b7884-srl99\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.370596 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.370496    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:03.481490 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.481446    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:03.540554 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:55:03.533316    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:55:03.540554 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.533885    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:03.839175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:03.838246    4839 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:04.142171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.141331    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 29 22:55:04.142171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.141394    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 29 22:55:04.174361 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.161382    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:55:04.197325 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.197253    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:04.295172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.291253    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:04.345243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.330325    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:55:04.356521 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.356492    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"]
Oct 29 22:55:04.361495 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.358997    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Oct 29 22:55:04.361495 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.359047    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Oct 29 22:55:04.602488 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.602443    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:55:04.613076 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.613041    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:55:04.682866 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:04.682840    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:55:05.185451 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:05.185399    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:05.466641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:05.466534    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99" event={"ID":"b1972eab-70dc-45f5-8f92-3061b0d862c4","Type":"ContainerStarted","Data":"fe47ec1b60f33ed788168fd893048d00d3118311206d9a775de1630e61f1b093"}
Oct 29 22:55:05.466641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:05.466585    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99" event={"ID":"b1972eab-70dc-45f5-8f92-3061b0d862c4","Type":"ContainerStarted","Data":"9b059bbbdf4cc6146f1071f24006cd59ae4ab37db5c4a8859ce212f6463f71cd"}
Oct 29 22:55:05.509468 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:05.509408    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99" podStartSLOduration=3.509389793 podStartE2EDuration="3.509389793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:05.508908254 +0000 UTC m=+762.611984518" watchObservedRunningTime="2026-10-29 22:55:05.509389793 +0000 UTC m=+762.612466066"
Oct 29 22:55:05.810864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:05.810818    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:55:05.811320 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:05.811292    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:55:05.812331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:05.812309    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 22:55:06.279658 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.279619    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:06.280212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.279696    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:06.342229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.342196    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:55:06.342569 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.342545    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:55:06.346625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.344244    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 22:55:06.477799 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.477762    4839 generic.go:356] "Generic (PLEG): container finished" podID="b1972eab-70dc-45f5-8f92-3061b0d862c4" containerID="fe47ec1b60f33ed788168fd893048d00d3118311206d9a775de1630e61f1b093" exitCode=0
Oct 29 22:55:06.477988 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.477845    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99" event={"ID":"b1972eab-70dc-45f5-8f92-3061b0d862c4","Type":"ContainerDied","Data":"fe47ec1b60f33ed788168fd893048d00d3118311206d9a775de1630e61f1b093"}
Oct 29 22:55:06.838586 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:06.838552    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:07.636901 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.636876    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:07.640448 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.640422    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1972eab-70dc-45f5-8f92-3061b0d862c4-kube-api-access-6ztwc\" (UniqueName: \"kubernetes.io/projected/b1972eab-70dc-45f5-8f92-3061b0d862c4-kube-api-access-6ztwc\") pod \"b1972eab-70dc-45f5-8f92-3061b0d862c4\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") "
Oct 29 22:55:07.640581 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.640552    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-xks-ca\") pod \"b1972eab-70dc-45f5-8f92-3061b0d862c4\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") "
Oct 29 22:55:07.640692 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.640678    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-openshift-ca\") pod \"b1972eab-70dc-45f5-8f92-3061b0d862c4\" (UID: \"b1972eab-70dc-45f5-8f92-3061b0d862c4\") "
Oct 29 22:55:07.640983 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.640962    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-openshift-ca" pod "b1972eab-70dc-45f5-8f92-3061b0d862c4" (UID: "b1972eab-70dc-45f5-8f92-3061b0d862c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:55:07.642477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.642457    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-xks-ca" pod "b1972eab-70dc-45f5-8f92-3061b0d862c4" (UID: "b1972eab-70dc-45f5-8f92-3061b0d862c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:55:07.642539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.642459    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1972eab-70dc-45f5-8f92-3061b0d862c4-kube-api-access-6ztwc" pod "b1972eab-70dc-45f5-8f92-3061b0d862c4" (UID: "b1972eab-70dc-45f5-8f92-3061b0d862c4"). InnerVolumeSpecName "kube-api-access-6ztwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:55:07.742553 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.742474    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:55:07.742553 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.742499    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztwc\" (UniqueName: \"kubernetes.io/projected/b1972eab-70dc-45f5-8f92-3061b0d862c4-kube-api-access-6ztwc\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:55:07.742553 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:07.742510    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1972eab-70dc-45f5-8f92-3061b0d862c4-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:55:08.432663 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.432625    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"]
Oct 29 22:55:08.433171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.433138    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b1972eab-70dc-45f5-8f92-3061b0d862c4" containerName="revoke-keys"
Oct 29 22:55:08.433262 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.433173    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1972eab-70dc-45f5-8f92-3061b0d862c4" containerName="revoke-keys"
Oct 29 22:55:08.433335 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.433324    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1972eab-70dc-45f5-8f92-3061b0d862c4" containerName="revoke-keys"
Oct 29 22:55:08.436784 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.436751    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.449613 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.449579    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.449821 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.449790    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7n2s\" (UniqueName: \"kubernetes.io/projected/5b0c4083-358e-4fd3-864b-db6b53a4990e-kube-api-access-b7n2s\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.449909 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.449892    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.449980 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.449925    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.450043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.449977    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.450043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.450035    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.450260 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.450244    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.450328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.450279    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.450412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.450384    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0c4083-358e-4fd3-864b-db6b53a4990e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.467134 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.467103    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"]
Oct 29 22:55:08.493823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.493794    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"
Oct 29 22:55:08.493996 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.493832    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99" event={"ID":"b1972eab-70dc-45f5-8f92-3061b0d862c4","Type":"ContainerDied","Data":"9b059bbbdf4cc6146f1071f24006cd59ae4ab37db5c4a8859ce212f6463f71cd"}
Oct 29 22:55:08.493996 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.493881    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b059bbbdf4cc6146f1071f24006cd59ae4ab37db5c4a8859ce212f6463f71cd"
Oct 29 22:55:08.512794 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.512765    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:08.551370 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551339    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0c4083-358e-4fd3-864b-db6b53a4990e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.551566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551391    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.551566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551423    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7n2s\" (UniqueName: \"kubernetes.io/projected/5b0c4083-358e-4fd3-864b-db6b53a4990e-kube-api-access-b7n2s\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.551566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551502    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.551566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551533    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551585    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551638    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551693    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.551727    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.552096    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.552136    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0c4083-358e-4fd3-864b-db6b53a4990e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.552375    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.552534    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.568514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.557032    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.578166 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.577579    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.579836 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.579756    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.591473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.590702    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0c4083-358e-4fd3-864b-db6b53a4990e-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.591473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.591283    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7n2s\" (UniqueName: \"kubernetes.io/projected/5b0c4083-358e-4fd3-864b-db6b53a4990e-kube-api-access-b7n2s\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-snxnt\" (UID: \"5b0c4083-358e-4fd3-864b-db6b53a4990e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:08.753413 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:08.753324    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:09.064867 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:55:09.064833    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0c4083_358e_4fd3_864b_db6b53a4990e.slice/crio-7271dd3c218ba79da49f5184781708ecac3fde7c9f4c8e096c6bcc4698a69076 WatchSource:0}: Error finding container 7271dd3c218ba79da49f5184781708ecac3fde7c9f4c8e096c6bcc4698a69076: Status 404 returned error can't find the container with id 7271dd3c218ba79da49f5184781708ecac3fde7c9f4c8e096c6bcc4698a69076
Oct 29 22:55:09.066058 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.066032    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"]
Oct 29 22:55:09.067398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.067363    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:09.067525 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.067428    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:09.067525 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.067460    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:09.502528 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.502439    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" event={"ID":"5b0c4083-358e-4fd3-864b-db6b53a4990e","Type":"ContainerStarted","Data":"f262fa6919a023beaac1694ffa2666f507caaef5ca25be90eef1fcbec4ebfe42"}
Oct 29 22:55:09.502528 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.502508    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" event={"ID":"5b0c4083-358e-4fd3-864b-db6b53a4990e","Type":"ContainerStarted","Data":"7271dd3c218ba79da49f5184781708ecac3fde7c9f4c8e096c6bcc4698a69076"}
Oct 29 22:55:09.567050 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.566989    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podStartSLOduration=1.566971218 podStartE2EDuration="1.566971218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:09.564634408 +0000 UTC m=+766.667710680" watchObservedRunningTime="2026-10-29 22:55:09.566971218 +0000 UTC m=+766.670047489"
Oct 29 22:55:09.754014 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:09.753932    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:10.066597 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:10.066558    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:10.755919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:10.755833    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:10.756590 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:10.755916    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:55:10.794914 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:10.794871    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:11.097326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.096909    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"]
Oct 29 22:55:11.101191 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.101143    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103161 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103115    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d24105-1239-4048-9d39-35c906573fc5-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103287 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103203    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103282    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103318    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d24105-1239-4048-9d39-35c906573fc5-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103355    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103376    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103431    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmrr\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-kube-api-access-lgmrr\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103490    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.103620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.103506    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.130085 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.130055    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"]
Oct 29 22:55:11.204457 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204413    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmrr\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-kube-api-access-lgmrr\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204483    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204501    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204528    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d24105-1239-4048-9d39-35c906573fc5-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204567    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204590    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204617    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d24105-1239-4048-9d39-35c906573fc5-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204648    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.204704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.204667    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.205068 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.205032    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.205131 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.205105    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.205444 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.205421    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.205549 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.205525    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d24105-1239-4048-9d39-35c906573fc5-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.205674 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.205523    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.207213 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.207188    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.207841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.207822    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d24105-1239-4048-9d39-35c906573fc5-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.226984 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.226957    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmrr\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-kube-api-access-lgmrr\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.227499 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.227477    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.415799 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.415704    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:11.751184 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:55:11.751133    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d24105_1239_4048_9d39_35c906573fc5.slice/crio-a54ae0a73b9f12a246b912c4183874d794da51ef539ca1ae0b6892a72e6cb051 WatchSource:0}: Error finding container a54ae0a73b9f12a246b912c4183874d794da51ef539ca1ae0b6892a72e6cb051: Status 404 returned error can't find the container with id a54ae0a73b9f12a246b912c4183874d794da51ef539ca1ae0b6892a72e6cb051
Oct 29 22:55:11.752971 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.752944    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"]
Oct 29 22:55:11.753817 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.753788    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:11.753900 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.753832    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:55:11.754199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.754167    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:11.754300 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.754244    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:11.754300 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.754283    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:11.983698 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:11.983661    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:12.532342 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.532306    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" event={"ID":"a5d24105-1239-4048-9d39-35c906573fc5","Type":"ContainerStarted","Data":"1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf"}
Oct 29 22:55:12.532527 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.532359    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" event={"ID":"a5d24105-1239-4048-9d39-35c906573fc5","Type":"ContainerStarted","Data":"a54ae0a73b9f12a246b912c4183874d794da51ef539ca1ae0b6892a72e6cb051"}
Oct 29 22:55:12.599180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.598504    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podStartSLOduration=1.598483675 podStartE2EDuration="1.598483675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:12.594420956 +0000 UTC m=+769.697497240" watchObservedRunningTime="2026-10-29 22:55:12.598483675 +0000 UTC m=+769.701559947"
Oct 29 22:55:12.754345 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.754312    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:12.754493 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.754361    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:55:12.829101 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.829022    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:55:12.840998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.840970    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:55:12.917169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:12.917084    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:13.418270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:13.418227    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:13.418804 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:13.418781    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:55:13.418932 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:13.418912    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:55:13.726389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:13.726299    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:13.754504 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:13.754474    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:13.754660 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:13.754547    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:55:14.418171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:14.418110    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:55:14.418356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:14.418235    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:55:14.754652 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:14.754552    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:14.754939 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:14.754892    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:55:14.805173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:14.803698    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:15.416650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:15.416612    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:55:15.416836 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:15.416683    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:55:15.517995 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:15.517955    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:15.754315 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:15.754225    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:15.754315 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:15.754294    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:55:16.051140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.051095    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"]
Oct 29 22:55:16.056956 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.056927    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.079455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.079042    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tmp-dir\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.079719 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.079696    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-model-cache\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.079948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.079929    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tls-certs\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.080080 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.080063    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-dshm\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.080234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.080216    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbjq\" (UniqueName: \"kubernetes.io/projected/6dd94a17-c06d-45cc-aeda-31c9b20457cc-kube-api-access-hxbjq\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.080505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.080487    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-home\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.086230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.081971    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-a2e7c8\"/\"distinct-w2-a2e7c8-kserve-self-signed-certs\""
Oct 29 22:55:16.093169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.089095    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"]
Oct 29 22:55:16.099845 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.099817    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:16.208196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.199164    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tls-certs\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.208494 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.208457    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-dshm\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.208613 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.208514    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbjq\" (UniqueName: \"kubernetes.io/projected/6dd94a17-c06d-45cc-aeda-31c9b20457cc-kube-api-access-hxbjq\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.208717 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.208695    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-home\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.208808 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.208791    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tmp-dir\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.208873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.208837    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-model-cache\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.209076 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.209053    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tls-certs\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.209346 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.209324    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-model-cache\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.209628 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.209607    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-home\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.209995 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.209968    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tmp-dir\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.211577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.211553    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-dshm\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.234109 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.234072    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbjq\" (UniqueName: \"kubernetes.io/projected/6dd94a17-c06d-45cc-aeda-31c9b20457cc-kube-api-access-hxbjq\") pod \"distinct-w2-a2e7c8-kserve-66d56fff79-77n7m\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") " pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.424252 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.420539    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:55:16.424252 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.420598    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:55:16.436640 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.436364    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:16.771878 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.771738    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 29 22:55:16.772384 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:16.772078    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 29 22:55:17.425960 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:17.422683    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:55:17.425960 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:17.422758    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:55:17.628172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:17.627636    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:17.646181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:17.636252    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:17.704064 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:17.703380    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:17.818219 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:17.816226    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"]
Oct 29 22:55:18.418123 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.418083    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 29 22:55:18.418363 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.418140    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 29 22:55:18.617292 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.617235    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" event={"ID":"6dd94a17-c06d-45cc-aeda-31c9b20457cc","Type":"ContainerStarted","Data":"7ce5ca1063c87859f34961c77b509a5ae5924be02c6983ccc38ce4176c56cf33"}
Oct 29 22:55:18.617493 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.617300    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" event={"ID":"6dd94a17-c06d-45cc-aeda-31c9b20457cc","Type":"ContainerStarted","Data":"b0951d3700c96a8d698478757cb953729939c1908f45a8ad64349d9af9aae63d"}
Oct 29 22:55:18.665213 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.665109    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podStartSLOduration=2.665089397 podStartE2EDuration="2.665089397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:18.663810793 +0000 UTC m=+775.766887065" watchObservedRunningTime="2026-10-29 22:55:18.665089397 +0000 UTC m=+775.768165670"
Oct 29 22:55:18.762708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.762599    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:55:18.762708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.762666    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:55:18.858197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.858164    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:18.859732 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.859704    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:18.861350 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:18.861322    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt"
Oct 29 22:55:19.200461 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:19.200427    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:19.437983 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:19.437948    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:19.656212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:19.656179    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:19.657086 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:19.657062    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:19.657903 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:19.657880    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:55:20.886455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:20.886416    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:22.467361 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:22.467328    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:23.957290 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:23.957248    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:25.612180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:25.609436    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:26.442461 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:26.441164    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:26.497442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:26.497410    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:26.772228 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:26.760175    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:26.822864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:26.822835    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:55:27.562025 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:27.561990    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:29.070411 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:29.070373    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:30.661343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:30.661301    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:32.153196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:32.153136    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:33.238293 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:55:33.238249    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:55:33.771639 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:33.771607    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:35.492947 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:35.492910    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:37.279391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:37.279350    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:38.956261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:38.956227    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:40.507514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:40.507476    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:41.146746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.146706    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"]
Oct 29 22:55:41.151515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.151488    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.178264 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.178233    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"]
Oct 29 22:55:41.179261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.179241    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-dd3ff3\"/\"distinct-2-w4-dd3ff3-kserve-self-signed-certs\""
Oct 29 22:55:41.193919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.193892    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:41.215065 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.215033    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-home\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.215237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.215069    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-dshm\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.215237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.215215    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-model-cache\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.215333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.215274    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp2k\" (UniqueName: \"kubernetes.io/projected/6d006a24-6341-4d53-b847-e49793bd0531-kube-api-access-nqp2k\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.215428 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.215409    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d006a24-6341-4d53-b847-e49793bd0531-tls-certs\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.215472 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.215438    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-tmp-dir\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316293 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316255    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d006a24-6341-4d53-b847-e49793bd0531-tls-certs\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316293 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316294    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-tmp-dir\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316555 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316342    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-home\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316555 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316359    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-dshm\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316555 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316416    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-model-cache\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316555 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316481    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp2k\" (UniqueName: \"kubernetes.io/projected/6d006a24-6341-4d53-b847-e49793bd0531-kube-api-access-nqp2k\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316839    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-home\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316934 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316872    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-tmp-dir\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.316934 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.316917    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-model-cache\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.318692 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.318657    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-dshm\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.318984 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.318965    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d006a24-6341-4d53-b847-e49793bd0531-tls-certs\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.382208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.379320    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp2k\" (UniqueName: \"kubernetes.io/projected/6d006a24-6341-4d53-b847-e49793bd0531-kube-api-access-nqp2k\") pod \"distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") " pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.476573 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.476483    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:41.779999 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:55:41.779961    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d006a24_6341_4d53_b847_e49793bd0531.slice/crio-e00c4041287dcfd548a5d4fb25ebd5283f6a969a4322ab7779910d9c7ca93acf WatchSource:0}: Error finding container e00c4041287dcfd548a5d4fb25ebd5283f6a969a4322ab7779910d9c7ca93acf: Status 404 returned error can't find the container with id e00c4041287dcfd548a5d4fb25ebd5283f6a969a4322ab7779910d9c7ca93acf
Oct 29 22:55:41.781522 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.781497    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"]
Oct 29 22:55:41.859385 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.859351    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" event={"ID":"6d006a24-6341-4d53-b847-e49793bd0531","Type":"ContainerStarted","Data":"66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b"}
Oct 29 22:55:41.859518 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.859398    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" event={"ID":"6d006a24-6341-4d53-b847-e49793bd0531","Type":"ContainerStarted","Data":"e00c4041287dcfd548a5d4fb25ebd5283f6a969a4322ab7779910d9c7ca93acf"}
Oct 29 22:55:41.906811 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:41.906752    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podStartSLOduration=0.906738596 podStartE2EDuration="906.738596ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:41.902974689 +0000 UTC m=+799.006050960" watchObservedRunningTime="2026-10-29 22:55:41.906738596 +0000 UTC m=+799.009814923"
Oct 29 22:55:42.434493 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:42.434462    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:42.476954 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:42.476918    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:44.240309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:44.240282    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:45.314384 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.314351    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"]
Oct 29 22:55:45.318411 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.318389    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.339180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.339131    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"]
Oct 29 22:55:45.348376 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.348351    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-81f92d\"/\"distinct-2-w6-81f92d-kserve-self-signed-certs\""
Oct 29 22:55:45.373431 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.373395    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:45.468979 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.468939    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-dshm\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.469175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.469003    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-model-cache\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.469175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.469094    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98q4\" (UniqueName: \"kubernetes.io/projected/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-kube-api-access-c98q4\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.469175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.469156    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tls-certs\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.469284 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.469259    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tmp-dir\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.469378 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.469364    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-home\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.573625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.573512    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-home\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.573819 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.573053    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-home\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.573979 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.573963    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-dshm\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.574450 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.574429    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-model-cache\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.574630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.574614    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98q4\" (UniqueName: \"kubernetes.io/projected/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-kube-api-access-c98q4\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.574792 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.574777    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tls-certs\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.575025 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.575010    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tmp-dir\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.600171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.575478    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tmp-dir\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.600419 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.577552    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-model-cache\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.600520 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.581996    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tls-certs\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.600590 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.598470    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-dshm\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.633964 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.633900    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98q4\" (UniqueName: \"kubernetes.io/projected/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-kube-api-access-c98q4\") pod \"distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") " pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.638794 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.638497    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:45.967943 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:55:45.967914    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3f0c9cf_92b3_4b5f_9818_1643ce63672f.slice/crio-a666cee1778e2495f51f9ac4d54d0c069396b998a55a7fe1c5080229c2168274 WatchSource:0}: Error finding container a666cee1778e2495f51f9ac4d54d0c069396b998a55a7fe1c5080229c2168274: Status 404 returned error can't find the container with id a666cee1778e2495f51f9ac4d54d0c069396b998a55a7fe1c5080229c2168274
Oct 29 22:55:45.969346 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:45.969322    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"]
Oct 29 22:55:46.641746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:46.641713    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:46.890999 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:46.890950    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" event={"ID":"e3f0c9cf-92b3-4b5f-9818-1643ce63672f","Type":"ContainerStarted","Data":"aa96d0207fde70f2936a0f082c9c02fe7b55c57a664c2f97749782d982d4c12d"}
Oct 29 22:55:46.890999 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:46.890998    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" event={"ID":"e3f0c9cf-92b3-4b5f-9818-1643ce63672f","Type":"ContainerStarted","Data":"a666cee1778e2495f51f9ac4d54d0c069396b998a55a7fe1c5080229c2168274"}
Oct 29 22:55:46.937833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:46.937726    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podStartSLOduration=1.937709691 podStartE2EDuration="1.937709691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:46.934711308 +0000 UTC m=+804.037787578" watchObservedRunningTime="2026-10-29 22:55:46.937709691 +0000 UTC m=+804.040785961"
Oct 29 22:55:47.640302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:47.640270    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:48.390791 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:48.390746    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:50.188123 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:50.188070    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:51.477486 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:51.477446    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:51.490175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:51.490131    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:51.736180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:51.736058    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:51.933058 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:51.933023    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:55:52.924955 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:52.924923    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:53.530443 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:53.530402    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:55.160201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.160167    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:55.487692 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.487612    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"]
Oct 29 22:55:55.491781 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.491763    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.513314 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.513285    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"]
Oct 29 22:55:55.606398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606361    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zcxg\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-kube-api-access-4zcxg\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606596 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606406    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606596 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606502    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606621    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7929816-c979-45d3-8dda-4688e78bc292-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606691    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606739    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606813    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606940 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606874    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.606940 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.606903    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7929816-c979-45d3-8dda-4688e78bc292-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.641205 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.641177    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:55.654107 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.654082    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:55.708379 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708344    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7929816-c979-45d3-8dda-4688e78bc292-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708420    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708456    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708494    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708549    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708573    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7929816-c979-45d3-8dda-4688e78bc292-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708647    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zcxg\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-kube-api-access-4zcxg\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708685    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.708864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708730    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.709018 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.708917    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.709103 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.709077    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.709405 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.709368    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.709518 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.709470    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.710058 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.710034    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7929816-c979-45d3-8dda-4688e78bc292-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.712388 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.712362    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.733270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.725893    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7929816-c979-45d3-8dda-4688e78bc292-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.762304 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.759758    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.771805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.763820    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zcxg\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-kube-api-access-4zcxg\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.812168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.808402    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:55.988566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:55.988164    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:55:56.037809 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.037766    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"]
Oct 29 22:55:56.050426 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.050393    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.079965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.079932    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"]
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119167    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119264    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119314    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119364    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119439    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktk5\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-kube-api-access-pktk5\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119487    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119555    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119588    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2821795-ad5a-4f7c-b0ea-080a8437f717-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.120182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.119631    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.186171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.185119    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"]
Oct 29 22:55:56.201196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.200103    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"]
Oct 29 22:55:56.201196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.200238    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.217223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.216781    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"]
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220434    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220522    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktk5\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-kube-api-access-pktk5\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220571    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220622    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220652    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2821795-ad5a-4f7c-b0ea-080a8437f717-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220691    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220822    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220851    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.220891    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.221770 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.221412    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:56.221770 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.221494    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:56.221770 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.221541    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:56.222426 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.222054    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.228463 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.224901    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2821795-ad5a-4f7c-b0ea-080a8437f717-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.228463 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.225453    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.228463 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.225746    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.228463 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.226474    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.230870 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.229831    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.235785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.235717    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.255523 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.255488    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktk5\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-kube-api-access-pktk5\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.257022 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.256991    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.322565    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.322662    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.322826    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.322884    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.323027    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.323115    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl59j\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-kube-api-access-dl59j\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.323177    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.323207    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.325188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.323268    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.402818 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.397850    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424580    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424638    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424722    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424789    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl59j\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-kube-api-access-dl59j\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424822    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424846    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424890    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.424964    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.425193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.425008    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.426186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.426087    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.428407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.428070    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.428407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.428289    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.428841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.428615    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.429089 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.429043    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.431305 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.431260    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.431635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.431610    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.454412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.454374    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.458447 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.457777    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl59j\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-kube-api-access-dl59j\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.534035 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.533774    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:56.831803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.831763    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:56.832662 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.832630    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:56.833763 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.833735    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:56.898491 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:56.898062    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"]
Oct 29 22:55:57.129657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.129585    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podStartSLOduration=2.12956179 podStartE2EDuration="2.12956179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:57.112545522 +0000 UTC m=+814.215621807" watchObservedRunningTime="2026-10-29 22:55:57.12956179 +0000 UTC m=+814.232638063"
Oct 29 22:55:57.333726 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:55:57.333639    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad61179b_7e0c_4bde_a7ee_e8272ae300b7.slice/crio-1825688121df46d9725fd0a9fec3fa666198811cd595c4b781d188325ab9c150 WatchSource:0}: Error finding container 1825688121df46d9725fd0a9fec3fa666198811cd595c4b781d188325ab9c150: Status 404 returned error can't find the container with id 1825688121df46d9725fd0a9fec3fa666198811cd595c4b781d188325ab9c150
Oct 29 22:55:57.366734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.364431    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:57.366734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.364528    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:57.366734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.364576    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:55:57.444580 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.444533    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" event={"ID":"d7929816-c979-45d3-8dda-4688e78bc292","Type":"ContainerStarted","Data":"65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9"}
Oct 29 22:55:57.445301 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.445268    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" event={"ID":"d7929816-c979-45d3-8dda-4688e78bc292","Type":"ContainerStarted","Data":"a986a489f016131c72b2b82366d4d2f217b5c0f6151a12db27d5630b0e1964e4"}
Oct 29 22:55:57.445696 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.445674    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"]
Oct 29 22:55:57.450300 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.450264    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" event={"ID":"d2821795-ad5a-4f7c-b0ea-080a8437f717","Type":"ContainerStarted","Data":"34113bb7c2fcf974c985c428b2b1ec1a8483c95a337215a9eec7824309295868"}
Oct 29 22:55:57.452654 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.452579    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:55:57.825348 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.825235    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:55:57.825727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.825697    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:55:57.825923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:57.825897    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:55:58.049671 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.049622    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" event={"ID":"ad61179b-7e0c-4bde-a7ee-e8272ae300b7","Type":"ContainerStarted","Data":"c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404"}
Oct 29 22:55:58.049846 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.049687    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" event={"ID":"ad61179b-7e0c-4bde-a7ee-e8272ae300b7","Type":"ContainerStarted","Data":"1825688121df46d9725fd0a9fec3fa666198811cd595c4b781d188325ab9c150"}
Oct 29 22:55:58.051710 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.051675    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" event={"ID":"d2821795-ad5a-4f7c-b0ea-080a8437f717","Type":"ContainerStarted","Data":"7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59"}
Oct 29 22:55:58.109168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.109098    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podStartSLOduration=2.109079556 podStartE2EDuration="2.109079556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:58.105439896 +0000 UTC m=+815.208516167" watchObservedRunningTime="2026-10-29 22:55:58.109079556 +0000 UTC m=+815.212155828"
Oct 29 22:55:58.155605 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.155539    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podStartSLOduration=2.155525408 podStartE2EDuration="2.155525408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:55:58.152273254 +0000 UTC m=+815.255349528" watchObservedRunningTime="2026-10-29 22:55:58.155525408 +0000 UTC m=+815.258601680"
Oct 29 22:55:58.398682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.398587    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:55:58.400404 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.400345    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:55:58.400535 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.400408    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:55:58.536197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.536157    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:55:58.537963 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.537934    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:55:58.538090 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.537985    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:55:58.816176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.812419    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:55:58.816176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:58.812483    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:55:59.398626 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:59.398589    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:55:59.398922 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:59.398887    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:55:59.537170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:59.536609    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:55:59.537170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:59.536678    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:55:59.811171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:59.809618    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:55:59.811171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:55:59.809684    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:56:00.161233 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.160212    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:00.258229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.258190    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:00.404295 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.400562    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.404295 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.400627    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:56:00.544919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.544559    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.544919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.544625    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:56:00.810229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.809543    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:00.810229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:00.809611    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:56:01.402179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:01.401555    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:01.402179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:01.401643    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:56:01.537171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:01.536532    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:56:01.537171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:01.536601    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:56:01.813171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:01.812585    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:01.813171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:01.812649    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:56:02.402295 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:02.402255    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.402553 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:02.402521    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:56:02.538172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:02.537526    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.538172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:02.537589    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:56:02.802051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:02.802015    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:02.825601 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:02.825560    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:02.827985 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:02.827949    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:56:03.248419 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:56:03.248329    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:56:03.404172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:03.399567    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:03.404172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:03.399634    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:56:03.536337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:03.536301    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:56:03.536492 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:03.536357    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:56:03.814699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:03.814601    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 29 22:56:03.815305 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:03.815272    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 29 22:56:04.401800 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:04.401767    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:04.402058 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:04.402031    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:56:04.536593 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:04.536548    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 29 22:56:04.536808 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:04.536627    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 29 22:56:04.995137 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:04.995100    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:56:04.996666 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:04.996636    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:56:05.000179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:04.999923    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 22:56:05.100292 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:05.100262    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:05.399234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:05.398373    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 29 22:56:05.399234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:05.398443    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 29 22:56:05.541960 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:05.541928    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:56:05.542397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:05.542370    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:56:05.543264 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:05.543243    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 22:56:06.403238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:06.403207    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:56:06.403631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:06.403488    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:56:06.404263 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:06.404244    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 22:56:06.800876 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:06.800845    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:08.455480 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:08.455441    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:10.073719 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:10.073683    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:11.710703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:11.710668    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:12.838642 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:12.838603    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:13.561134 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:13.561098    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:14.934889 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:14.934855    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:15.220188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:15.220074    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:17.066103 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:17.066004    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:18.067238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:18.067192    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:18.324047 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:18.323942    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:18.627230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:18.626253    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:18.631171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:18.628380    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:56:18.691573 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:18.691522    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:56:18.754863 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:18.754813    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:19.491338 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:19.491304    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:20.997635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:20.997603    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:22.791525 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:22.791493    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:24.367124 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:24.367091    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:25.957932 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:25.957887    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:26.323483 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.323446    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"]
Oct 29 22:56:26.328728 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.328705    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.344703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.344672    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"]
Oct 29 22:56:26.351341 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.351314    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.351491 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.351386    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.351564 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.351519    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvmq\" (UniqueName: \"kubernetes.io/projected/43f12c21-22f6-4917-b443-2c1caaa6c6b0-kube-api-access-klvmq\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.354208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.354180    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:26.354323 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.354224    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:56:26.443684 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.443647    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"]
Oct 29 22:56:26.447919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.447888    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.452814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.452781    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfefde5-a0f8-41ce-ab36-39799b074499-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.452975 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.452852    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.452975 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.452956    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453063 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.452997    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453103 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453073    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.453135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453121    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453280 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453260    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453305    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453451 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453414    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnm67\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-kube-api-access-qnm67\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453544 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453477    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453612 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453565    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvmq\" (UniqueName: \"kubernetes.io/projected/43f12c21-22f6-4917-b443-2c1caaa6c6b0-kube-api-access-klvmq\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.453666 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453617    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.453844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.453827    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.455516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.455402    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.465176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.465133    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"]
Oct 29 22:56:26.478368 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.478330    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"]
Oct 29 22:56:26.478523 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.478484    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.480085 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.480056    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvmq\" (UniqueName: \"kubernetes.io/projected/43f12c21-22f6-4917-b443-2c1caaa6c6b0-kube-api-access-klvmq\") pod \"maas-api-revoke-sub-5f9d3a66-hdmlz\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.493663 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.493636    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"]
Oct 29 22:56:26.554388 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554347    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.554388 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554395    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.554641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554419    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.554641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554458    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnm67\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-kube-api-access-qnm67\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.554641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554475    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.554641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554508    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.554858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554713    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.554858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554756    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.554858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554769    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.554858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554818    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.555013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554864    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfefde5-a0f8-41ce-ab36-39799b074499-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.555013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554907    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.555013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.554927    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.555159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555017    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.555159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555043    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.555159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555063    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.555159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555083    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.555159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555135    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.555400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555212    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.555400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555323    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8mc9\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-kube-api-access-d8mc9\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.555522 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555501    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.555560 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555507    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.555684 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.555668    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfefde5-a0f8-41ce-ab36-39799b074499-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.556994 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.556971    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.557438 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.557422    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.575873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.575813    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.576427 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.576406    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnm67\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-kube-api-access-qnm67\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.645853 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.645816    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:26.656698 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.656669    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.656842 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.656723    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.656842 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.656759    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.656951 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.656857    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.656951 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.656887    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657053 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.656960    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657053 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.656992    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657053 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.657017    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657053 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.657046    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8mc9\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-kube-api-access-d8mc9\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.657231    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-data\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.657265    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657460 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.657434    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-certs\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657567 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.657501    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-credential-socket\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.657856 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.657838    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istiod-ca-cert\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.659611 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.659593    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-envoy\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.659865 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.659845    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-podinfo\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.681275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.681248    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-token\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.681426 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.681280    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8mc9\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-kube-api-access-d8mc9\") pod \"e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") " pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.784443 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.784411    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:26.812568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.812539    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:26.947474 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:56:26.947443    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f12c21_22f6_4917_b443_2c1caaa6c6b0.slice/crio-b08bb1a3fd3e8310121e9c305d12ce1950e9c771a8836b709de8e60b0692085d WatchSource:0}: Error finding container b08bb1a3fd3e8310121e9c305d12ce1950e9c771a8836b709de8e60b0692085d: Status 404 returned error can't find the container with id b08bb1a3fd3e8310121e9c305d12ce1950e9c771a8836b709de8e60b0692085d
Oct 29 22:56:26.948915 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:26.948889    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"]
Oct 29 22:56:27.085649 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:56:27.085619    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dfefde5_a0f8_41ce_ab36_39799b074499.slice/crio-478d7eef837124d400709b6cd2eb99403ff6620cbf9a9cf3548a654fc84530b7 WatchSource:0}: Error finding container 478d7eef837124d400709b6cd2eb99403ff6620cbf9a9cf3548a654fc84530b7: Status 404 returned error can't find the container with id 478d7eef837124d400709b6cd2eb99403ff6620cbf9a9cf3548a654fc84530b7
Oct 29 22:56:27.088261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.088227    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:27.088355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.088296    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:27.088355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.088328    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:27.089070 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.089046    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"]
Oct 29 22:56:27.129520 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:56:27.129494    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff42ba4_c22f_4659_9dc9_cd039d8b569f.slice/crio-b37ba5dae006e2d70eb2be79c8c042561732496441d7f10cde493d25e3c14b29 WatchSource:0}: Error finding container b37ba5dae006e2d70eb2be79c8c042561732496441d7f10cde493d25e3c14b29: Status 404 returned error can't find the container with id b37ba5dae006e2d70eb2be79c8c042561732496441d7f10cde493d25e3c14b29
Oct 29 22:56:27.130620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.130594    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"]
Oct 29 22:56:27.132665 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.132635    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:27.132746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.132699    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:27.132746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.132728    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:27.588006 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.587951    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" event={"ID":"8dfefde5-a0f8-41ce-ab36-39799b074499","Type":"ContainerStarted","Data":"37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad"}
Oct 29 22:56:27.588258 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.588081    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" event={"ID":"8dfefde5-a0f8-41ce-ab36-39799b074499","Type":"ContainerStarted","Data":"478d7eef837124d400709b6cd2eb99403ff6620cbf9a9cf3548a654fc84530b7"}
Oct 29 22:56:27.590020 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.589993    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" event={"ID":"eff42ba4-c22f-4659-9dc9-cd039d8b569f","Type":"ContainerStarted","Data":"cb3c56cc7568a2220b810f578bae6379e3d3a6f061a3acc8f5209c96703c3eb6"}
Oct 29 22:56:27.590200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.590043    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" event={"ID":"eff42ba4-c22f-4659-9dc9-cd039d8b569f","Type":"ContainerStarted","Data":"b37ba5dae006e2d70eb2be79c8c042561732496441d7f10cde493d25e3c14b29"}
Oct 29 22:56:27.591982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.591952    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz" event={"ID":"43f12c21-22f6-4917-b443-2c1caaa6c6b0","Type":"ContainerStarted","Data":"7915112f2e232ce52b6af4267398159dd1485780598390ec7310bcff84accad0"}
Oct 29 22:56:27.592090 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.592005    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz" event={"ID":"43f12c21-22f6-4917-b443-2c1caaa6c6b0","Type":"ContainerStarted","Data":"b08bb1a3fd3e8310121e9c305d12ce1950e9c771a8836b709de8e60b0692085d"}
Oct 29 22:56:27.643175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.642457    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podStartSLOduration=1.642438801 podStartE2EDuration="1.642438801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:27.639508293 +0000 UTC m=+844.742584565" watchObservedRunningTime="2026-10-29 22:56:27.642438801 +0000 UTC m=+844.745515074"
Oct 29 22:56:27.670610 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.668412    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:27.697854 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.697763    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz" podStartSLOduration=1.697740499 podStartE2EDuration="1.697740499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:27.682616588 +0000 UTC m=+844.785692861" watchObservedRunningTime="2026-10-29 22:56:27.697740499 +0000 UTC m=+844.800816771"
Oct 29 22:56:27.789173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.787283    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:27.833892 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:27.833850    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:28.437681 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.437641    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:28.704532 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.704441    4839 generic.go:356] "Generic (PLEG): container finished" podID="43f12c21-22f6-4917-b443-2c1caaa6c6b0" containerID="7915112f2e232ce52b6af4267398159dd1485780598390ec7310bcff84accad0" exitCode=0
Oct 29 22:56:28.714181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.704925    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz" event={"ID":"43f12c21-22f6-4917-b443-2c1caaa6c6b0","Type":"ContainerDied","Data":"7915112f2e232ce52b6af4267398159dd1485780598390ec7310bcff84accad0"}
Oct 29 22:56:28.794170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.791701    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:56:28.794170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.791791    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:56:28.799191 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.798902    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podStartSLOduration=2.798880786 podStartE2EDuration="2.798880786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:27.755584189 +0000 UTC m=+844.858660462" watchObservedRunningTime="2026-10-29 22:56:28.798880786 +0000 UTC m=+845.901957077"
Oct 29 22:56:28.865171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.864801    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:56:28.865171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:28.864862    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 29 22:56:29.734831 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:29.734794    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:29.794174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:29.793548    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:56:29.794174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:29.793611    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:56:29.843171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:29.814156    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:56:29.843171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:29.814219    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 29 22:56:30.253556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.253527    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:30.359629 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.359594    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-openshift-ca\") pod \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") "
Oct 29 22:56:30.359776 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.359734    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-xks-ca\") pod \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") "
Oct 29 22:56:30.359855 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.359840    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f12c21-22f6-4917-b443-2c1caaa6c6b0-kube-api-access-klvmq\" (UniqueName: \"kubernetes.io/projected/43f12c21-22f6-4917-b443-2c1caaa6c6b0-kube-api-access-klvmq\") pod \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\" (UID: \"43f12c21-22f6-4917-b443-2c1caaa6c6b0\") "
Oct 29 22:56:30.360008 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.359965    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-openshift-ca" pod "43f12c21-22f6-4917-b443-2c1caaa6c6b0" (UID: "43f12c21-22f6-4917-b443-2c1caaa6c6b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:30.360478 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.360455    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:30.378171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.363838    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f12c21-22f6-4917-b443-2c1caaa6c6b0-kube-api-access-klvmq" pod "43f12c21-22f6-4917-b443-2c1caaa6c6b0" (UID: "43f12c21-22f6-4917-b443-2c1caaa6c6b0"). InnerVolumeSpecName "kube-api-access-klvmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:30.378171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.364326    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-xks-ca" pod "43f12c21-22f6-4917-b443-2c1caaa6c6b0" (UID: "43f12c21-22f6-4917-b443-2c1caaa6c6b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:30.462289 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.462246    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klvmq\" (UniqueName: \"kubernetes.io/projected/43f12c21-22f6-4917-b443-2c1caaa6c6b0-kube-api-access-klvmq\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:30.462289 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.462279    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f12c21-22f6-4917-b443-2c1caaa6c6b0-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:30.784172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.775056    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz" event={"ID":"43f12c21-22f6-4917-b443-2c1caaa6c6b0","Type":"ContainerDied","Data":"b08bb1a3fd3e8310121e9c305d12ce1950e9c771a8836b709de8e60b0692085d"}
Oct 29 22:56:30.784172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.775127    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b08bb1a3fd3e8310121e9c305d12ce1950e9c771a8836b709de8e60b0692085d"
Oct 29 22:56:30.784172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.775221    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"
Oct 29 22:56:30.817186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.797705    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:56:30.817186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.797799    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:56:30.838199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.825071    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:56:30.838199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:30.825129    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 29 22:56:31.439580 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:31.439548    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:31.787173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:31.785274    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:56:31.787173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:31.785361    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:56:31.817170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:31.813473    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:56:31.817170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:31.813548    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 29 22:56:32.794172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:32.793250    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:56:32.794172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:32.793314    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:56:32.821308 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:32.820304    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:56:32.821308 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:32.820380    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 29 22:56:33.102944 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:33.102844    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:33.334343 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:56:33.334301    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:56:33.785786 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:33.785743    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 29 22:56:33.785978 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:33.785817    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 29 22:56:33.813501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:33.813467    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 29 22:56:33.813864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:33.813544    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 29 22:56:34.797493 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:34.797448    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:35.055407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:35.031114    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:35.055407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:35.034704    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:35.091162 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:35.091110    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:56:35.219042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:35.218573    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:35.219253 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:35.219203    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:35.220773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:35.220741    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:56:36.560391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:36.560350    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:37.887417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.887384    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"]
Oct 29 22:56:37.887923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.887909    4839 cpu_manager.go:414] "containerMap: removing container" podUID="43f12c21-22f6-4917-b443-2c1caaa6c6b0" containerName="revoke-keys"
Oct 29 22:56:37.887972 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.887924    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f12c21-22f6-4917-b443-2c1caaa6c6b0" containerName="revoke-keys"
Oct 29 22:56:37.888011 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.888001    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f12c21-22f6-4917-b443-2c1caaa6c6b0" containerName="revoke-keys"
Oct 29 22:56:37.891630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.891609    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:37.908766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.908738    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"]
Oct 29 22:56:37.918579 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.918550    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:37.918819 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:37.918803    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:56:38.010181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.010115    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.010376 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.010312    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.010376 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.010341    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnwj\" (UniqueName: \"kubernetes.io/projected/30708359-1b51-4b63-b917-ae800aa4ea68-kube-api-access-zbnwj\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.111929 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.111888    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.112135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.112016    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.112135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.112035    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnwj\" (UniqueName: \"kubernetes.io/projected/30708359-1b51-4b63-b917-ae800aa4ea68-kube-api-access-zbnwj\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.113408 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.113375    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.122201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.122172    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.161046 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.159060    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnwj\" (UniqueName: \"kubernetes.io/projected/30708359-1b51-4b63-b917-ae800aa4ea68-kube-api-access-zbnwj\") pod \"maas-api-revoke-sub-37dd5804-644qm\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.208727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.208393    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:38.221610 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.221573    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:38.515175 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:56:38.515134    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30708359_1b51_4b63_b917_ae800aa4ea68.slice/crio-b0fcb979896f5b3e1de3f474877acc6407675420aa29abc26be293c4ee9591ec WatchSource:0}: Error finding container b0fcb979896f5b3e1de3f474877acc6407675420aa29abc26be293c4ee9591ec: Status 404 returned error can't find the container with id b0fcb979896f5b3e1de3f474877acc6407675420aa29abc26be293c4ee9591ec
Oct 29 22:56:38.516265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.516237    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"]
Oct 29 22:56:38.875931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.875895    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm" event={"ID":"30708359-1b51-4b63-b917-ae800aa4ea68","Type":"ContainerStarted","Data":"442ea113b62fe009bde3af2f7b7d655a0468257edb7a88d9baee751f368fff4b"}
Oct 29 22:56:38.876115 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.875953    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm" event={"ID":"30708359-1b51-4b63-b917-ae800aa4ea68","Type":"ContainerStarted","Data":"b0fcb979896f5b3e1de3f474877acc6407675420aa29abc26be293c4ee9591ec"}
Oct 29 22:56:38.936071 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:38.936023    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm" podStartSLOduration=1.936006991 podStartE2EDuration="1.936006991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:38.919902024 +0000 UTC m=+856.022978300" watchObservedRunningTime="2026-10-29 22:56:38.936006991 +0000 UTC m=+856.039083268"
Oct 29 22:56:39.884715 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:39.884680    4839 generic.go:356] "Generic (PLEG): container finished" podID="30708359-1b51-4b63-b917-ae800aa4ea68" containerID="442ea113b62fe009bde3af2f7b7d655a0468257edb7a88d9baee751f368fff4b" exitCode=0
Oct 29 22:56:39.884893 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:39.884766    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm" event={"ID":"30708359-1b51-4b63-b917-ae800aa4ea68","Type":"ContainerDied","Data":"442ea113b62fe009bde3af2f7b7d655a0468257edb7a88d9baee751f368fff4b"}
Oct 29 22:56:39.938402 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:39.938358    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:41.049425 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.049399    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:41.155107 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.155024    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-xks-ca\") pod \"30708359-1b51-4b63-b917-ae800aa4ea68\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") "
Oct 29 22:56:41.155107 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.155065    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-openshift-ca\") pod \"30708359-1b51-4b63-b917-ae800aa4ea68\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") "
Oct 29 22:56:41.155309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.155242    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30708359-1b51-4b63-b917-ae800aa4ea68-kube-api-access-zbnwj\" (UniqueName: \"kubernetes.io/projected/30708359-1b51-4b63-b917-ae800aa4ea68-kube-api-access-zbnwj\") pod \"30708359-1b51-4b63-b917-ae800aa4ea68\" (UID: \"30708359-1b51-4b63-b917-ae800aa4ea68\") "
Oct 29 22:56:41.155518 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.155489    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-openshift-ca" pod "30708359-1b51-4b63-b917-ae800aa4ea68" (UID: "30708359-1b51-4b63-b917-ae800aa4ea68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:41.155839 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.155814    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:41.157372 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.157350    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30708359-1b51-4b63-b917-ae800aa4ea68-kube-api-access-zbnwj" pod "30708359-1b51-4b63-b917-ae800aa4ea68" (UID: "30708359-1b51-4b63-b917-ae800aa4ea68"). InnerVolumeSpecName "kube-api-access-zbnwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:41.157464 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.157392    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-xks-ca" pod "30708359-1b51-4b63-b917-ae800aa4ea68" (UID: "30708359-1b51-4b63-b917-ae800aa4ea68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:41.256844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.256810    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbnwj\" (UniqueName: \"kubernetes.io/projected/30708359-1b51-4b63-b917-ae800aa4ea68-kube-api-access-zbnwj\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:41.256844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.256840    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30708359-1b51-4b63-b917-ae800aa4ea68-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:41.269413 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.269381    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"]
Oct 29 22:56:41.269895 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.269882    4839 cpu_manager.go:414] "containerMap: removing container" podUID="30708359-1b51-4b63-b917-ae800aa4ea68" containerName="revoke-keys"
Oct 29 22:56:41.269937 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.269896    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30708359-1b51-4b63-b917-ae800aa4ea68" containerName="revoke-keys"
Oct 29 22:56:41.270009 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.270000    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="30708359-1b51-4b63-b917-ae800aa4ea68" containerName="revoke-keys"
Oct 29 22:56:41.273511 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.273492    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.297514 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.297484    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"]
Oct 29 22:56:41.358430 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358398    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358447    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358578    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358608    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358627    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sqnt\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-kube-api-access-4sqnt\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358740 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358674    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358774 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358758    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358788    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.358908 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.358892    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.460738 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460657    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.460738 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460696    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.460912 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460752    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.460912 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460841    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.460912 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460871    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.460912 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460902    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.461042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460917    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.461042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460932    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sqnt\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-kube-api-access-4sqnt\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.461042 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.460956    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.461763 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.461391    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.462127 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.462086    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-credential-socket\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.462290 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.462273    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-certs\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.462358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.462290    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-data\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.462596 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.462577    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istiod-ca-cert\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.464008 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.463990    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-envoy\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.464475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.464457    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-podinfo\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.482465 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.482434    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-token\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.482704 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.482674    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sqnt\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-kube-api-access-4sqnt\") pod \"e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") " pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.522963 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.522936    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:41.595712 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.595426    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:41.908297 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.908267    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"
Oct 29 22:56:41.908473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.908265    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm" event={"ID":"30708359-1b51-4b63-b917-ae800aa4ea68","Type":"ContainerDied","Data":"b0fcb979896f5b3e1de3f474877acc6407675420aa29abc26be293c4ee9591ec"}
Oct 29 22:56:41.908473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.908434    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0fcb979896f5b3e1de3f474877acc6407675420aa29abc26be293c4ee9591ec"
Oct 29 22:56:41.966659 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.966626    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"]
Oct 29 22:56:41.972360 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:56:41.972323    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cae13af_8c9f_4da1_9e48_5e22cfb39b42.slice/crio-2a15579b21cf5c578984cf57839a667aee02763b5b28a397fe7659cb9e6abfd5 WatchSource:0}: Error finding container 2a15579b21cf5c578984cf57839a667aee02763b5b28a397fe7659cb9e6abfd5: Status 404 returned error can't find the container with id 2a15579b21cf5c578984cf57839a667aee02763b5b28a397fe7659cb9e6abfd5
Oct 29 22:56:41.975084 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.975056    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:41.975179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.975122    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:41.975240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:41.975187    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:56:42.916810 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:42.916768    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" event={"ID":"6cae13af-8c9f-4da1-9e48-5e22cfb39b42","Type":"ContainerStarted","Data":"59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce"}
Oct 29 22:56:42.917251 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:42.916833    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" event={"ID":"6cae13af-8c9f-4da1-9e48-5e22cfb39b42","Type":"ContainerStarted","Data":"2a15579b21cf5c578984cf57839a667aee02763b5b28a397fe7659cb9e6abfd5"}
Oct 29 22:56:43.000937 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:43.000886    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podStartSLOduration=2.000872029 podStartE2EDuration="2.000872029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:56:42.985631403 +0000 UTC m=+860.088707675" watchObservedRunningTime="2026-10-29 22:56:43.000872029 +0000 UTC m=+860.103948302"
Oct 29 22:56:43.141703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:43.141673    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:43.596490 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:43.596448    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:43.598190 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:43.598162    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:56:43.598328 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:43.598212    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:56:44.597127 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:44.597090    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:56:44.597550 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:44.597196    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:56:45.114261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:45.114223    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:45.596736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:45.596695    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:56:45.596905 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:45.596767    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:56:46.597134 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:46.597094    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:56:46.597584 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:46.597170    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:56:46.827757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:46.827722    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:47.597119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:47.597084    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:56:47.597339 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:47.597141    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:56:48.373697 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:48.373661    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:48.596539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:48.596506    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 29 22:56:48.596722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:48.596560    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 29 22:56:48.659919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:48.659838    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:50.623934 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:50.623266    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:50.623934 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:50.623367    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:50.980062 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:50.979976    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:51.601440 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.601377    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:51.621449 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.621390    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.627468 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.627414    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.631179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.627434    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.694353 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.694316    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:51.695634 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.695202    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.697816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.697694    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:51.757025 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.751464    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.855235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.850607    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.952171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.948281    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:56:51.980303 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:51.980246    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.073832 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.073781    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:56:52.136728 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.136631    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"]
Oct 29 22:56:52.321172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.311465    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"]
Oct 29 22:56:52.346172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.315281    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.348158 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.328285    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.404288 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.396637    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:56:52.494471 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.494431    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:52.494740 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.494710    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.496503 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.496469    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 29 22:56:52.498584 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.496872    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:56:52.608850 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.608806    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:52.609128 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.609097    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.610630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.610598    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.610873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.610852    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.618394 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.618361    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kjv\" (UniqueName: \"kubernetes.io/projected/aa249b06-17bc-4da1-8bd2-3061084bb1cd-kube-api-access-96kjv\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.711469 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.673273    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.711469 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.673615    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:56:52.711469 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.673861    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:56:52.711469 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.673918    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:56:52.711469 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.674824    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:52.711469 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.674878    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.772838 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.772792    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.773210 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.773182    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.773403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.773384    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kjv\" (UniqueName: \"kubernetes.io/projected/aa249b06-17bc-4da1-8bd2-3061084bb1cd-kube-api-access-96kjv\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.785954 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.785918    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:52.916173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.798530    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.916173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.798572    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.916173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.895710    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:56:52.916173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.809613    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:52.916173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.895788    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.916173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.895962    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:56:52.916173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.856349    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:52.930176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.922792    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:56:52.961884 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:52.961601    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.015912 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.015878    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:56:53.045027 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.008922    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.094174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.073588    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.111459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.103680    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:56:53.197171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.193122    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kjv\" (UniqueName: \"kubernetes.io/projected/aa249b06-17bc-4da1-8bd2-3061084bb1cd-kube-api-access-96kjv\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:53.233441 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.233352    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d48892b-tz4wk\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:53.362181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.349942    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.546518 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.546484    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:56:53.546897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.546876    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:56:53.547096 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.547081    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:56:53.547249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.547236    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:56:53.564383 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.564347    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:53.593128 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.593092    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:53.674344 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.674300    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:53.686319 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.686280    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:56:53.967130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.951335    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:56:53.967130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.951884    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:56:53.970265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:53.969066    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.031194 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.031121    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:56:54.110331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.110287    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.125193 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.124904    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.173790 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.173743    4839 patch_prober.go:29] interesting pod/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:56:54.174058 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.174028    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:56:54.207170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.201016    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:56:54.207170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.201106    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:56:54.207170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.201177    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:56:54.463635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.462595    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:56:54.549476 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.549434    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:56:54.550897 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.550860    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:56:54.601516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.601482    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:54.671257 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.671060    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:56:54.673385 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.673361    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:56:54.681973 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.681935    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:54.705196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.702519    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 22:56:54.713630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.713598    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:56:54.804309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.803184    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"]
Oct 29 22:56:54.805213 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:56:54.805171    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa249b06_17bc_4da1_8bd2_3061084bb1cd.slice/crio-6b09cc2df791277ba594c6a51d2b67cf83b6615dcd0c43672a1f8fd84a0e8c49 WatchSource:0}: Error finding container 6b09cc2df791277ba594c6a51d2b67cf83b6615dcd0c43672a1f8fd84a0e8c49: Status 404 returned error can't find the container with id 6b09cc2df791277ba594c6a51d2b67cf83b6615dcd0c43672a1f8fd84a0e8c49
Oct 29 22:56:54.884397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:54.884358    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:55.743830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:55.743791    4839 generic.go:356] "Generic (PLEG): container finished" podID="aa249b06-17bc-4da1-8bd2-3061084bb1cd" containerID="0935b5814e3e7751756bddbdc8b9c652f629f4db1aeb49637ede51fb456d4061" exitCode=0
Oct 29 22:56:55.744255 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:55.743876    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk" event={"ID":"aa249b06-17bc-4da1-8bd2-3061084bb1cd","Type":"ContainerDied","Data":"0935b5814e3e7751756bddbdc8b9c652f629f4db1aeb49637ede51fb456d4061"}
Oct 29 22:56:55.744255 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:55.743941    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk" event={"ID":"aa249b06-17bc-4da1-8bd2-3061084bb1cd","Type":"ContainerStarted","Data":"6b09cc2df791277ba594c6a51d2b67cf83b6615dcd0c43672a1f8fd84a0e8c49"}
Oct 29 22:56:56.681563 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:56.681534    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:56:56.993062 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:56.993037    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:57.075075 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.075037    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-xks-ca\") pod \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") "
Oct 29 22:56:57.075265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.075081    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-openshift-ca\") pod \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") "
Oct 29 22:56:57.075325 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.075269    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa249b06-17bc-4da1-8bd2-3061084bb1cd-kube-api-access-96kjv\" (UniqueName: \"kubernetes.io/projected/aa249b06-17bc-4da1-8bd2-3061084bb1cd-kube-api-access-96kjv\") pod \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\" (UID: \"aa249b06-17bc-4da1-8bd2-3061084bb1cd\") "
Oct 29 22:56:57.075525 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.075501    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-openshift-ca" pod "aa249b06-17bc-4da1-8bd2-3061084bb1cd" (UID: "aa249b06-17bc-4da1-8bd2-3061084bb1cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:56:57.075780 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.075764    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:57.077240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.077220    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-xks-ca" pod "aa249b06-17bc-4da1-8bd2-3061084bb1cd" (UID: "aa249b06-17bc-4da1-8bd2-3061084bb1cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:56:57.077316 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.077259    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa249b06-17bc-4da1-8bd2-3061084bb1cd-kube-api-access-96kjv" pod "aa249b06-17bc-4da1-8bd2-3061084bb1cd" (UID: "aa249b06-17bc-4da1-8bd2-3061084bb1cd"). InnerVolumeSpecName "kube-api-access-96kjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:56:57.177416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.177379    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96kjv\" (UniqueName: \"kubernetes.io/projected/aa249b06-17bc-4da1-8bd2-3061084bb1cd-kube-api-access-96kjv\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:57.177416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.177410    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa249b06-17bc-4da1-8bd2-3061084bb1cd-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:56:57.783815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.783783    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"
Oct 29 22:56:57.783989 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.783828    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk" event={"ID":"aa249b06-17bc-4da1-8bd2-3061084bb1cd","Type":"ContainerDied","Data":"6b09cc2df791277ba594c6a51d2b67cf83b6615dcd0c43672a1f8fd84a0e8c49"}
Oct 29 22:56:57.783989 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:57.783905    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b09cc2df791277ba594c6a51d2b67cf83b6615dcd0c43672a1f8fd84a0e8c49"
Oct 29 22:56:58.261599 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:56:58.261521    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:00.128585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:00.128552    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:01.755603 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:01.755554    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:03.239232 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:57:03.239195    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:57:03.415672 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:03.415609    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:05.203172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:05.200132    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:06.657172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:06.656301    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.680173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:06.679449    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.967453 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:06.967344    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:06.968645 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:06.967833    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:06.976266 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:06.976227    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.119809 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.119761    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.169164 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.144688    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.497341 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.497281    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": context deadline exceeded"
Oct 29 22:57:07.677301 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.677255    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.679782 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.679757    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:57:07.735169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.729266    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:07.735169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.729972    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:57:07.829223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.829193    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:07.843171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.841263    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:57:07.846118 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:07.846096    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:57:09.490100 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:09.490042    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:10.679922 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.679885    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"]
Oct 29 22:57:10.680436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.680422    4839 cpu_manager.go:414] "containerMap: removing container" podUID="aa249b06-17bc-4da1-8bd2-3061084bb1cd" containerName="revoke-keys"
Oct 29 22:57:10.680436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.680437    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa249b06-17bc-4da1-8bd2-3061084bb1cd" containerName="revoke-keys"
Oct 29 22:57:10.680561 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.680552    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa249b06-17bc-4da1-8bd2-3061084bb1cd" containerName="revoke-keys"
Oct 29 22:57:10.685627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.685595    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.709700 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.709671    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:10.723430 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.723405    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:10.738460 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.738429    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"]
Oct 29 22:57:10.745230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.745198    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.745434 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.745408    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.745512 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.745502    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjg2m\" (UniqueName: \"kubernetes.io/projected/219fa173-07df-42dc-9cb8-c5f5d2043fb0-kube-api-access-fjg2m\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.846684 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.846650    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.846886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.846714    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjg2m\" (UniqueName: \"kubernetes.io/projected/219fa173-07df-42dc-9cb8-c5f5d2043fb0-kube-api-access-fjg2m\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.846886 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.846840    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.847675 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.847649    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.849352 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.849331    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:10.885698 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:10.885665    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjg2m\" (UniqueName: \"kubernetes.io/projected/219fa173-07df-42dc-9cb8-c5f5d2043fb0-kube-api-access-fjg2m\") pod \"maas-api-revoke-sub-9977110e-rffhp\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:11.000416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.000337    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:11.218091 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.218054    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"]
Oct 29 22:57:11.222613 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.222588    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.238933 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.238902    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"]
Oct 29 22:57:11.242766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.242746    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.252579 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252490    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.252579 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252570    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.252814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252600    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.252814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252662    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.252814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252685    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.252814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252745    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.252814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252770    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0486bcec-d668-4e37-a171-d1ef285293e9-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.252814 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252793    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzc29\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-kube-api-access-xzc29\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252832    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252859    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252922    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.252982    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.253017    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0486bcec-d668-4e37-a171-d1ef285293e9-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.253053    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.253080    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.253113    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.253202    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrph2\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-kube-api-access-lrph2\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.253249    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.261805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.258608    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"]
Oct 29 22:57:11.299648 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.299616    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"]
Oct 29 22:57:11.320076 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.320046    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"]
Oct 29 22:57:11.320367 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:11.320345    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219fa173_07df_42dc_9cb8_c5f5d2043fb0.slice/crio-165755a906d3f8ecd6682765c03c62a9b3e8e391dede47d11ec5712fe127c4fd WatchSource:0}: Error finding container 165755a906d3f8ecd6682765c03c62a9b3e8e391dede47d11ec5712fe127c4fd: Status 404 returned error can't find the container with id 165755a906d3f8ecd6682765c03c62a9b3e8e391dede47d11ec5712fe127c4fd
Oct 29 22:57:11.354563 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.354529    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.354696 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.354579    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.354696 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.354639    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.354696 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.354665    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.354838 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.354719    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.354838 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.354746    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0486bcec-d668-4e37-a171-d1ef285293e9-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355008 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.354983    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355048 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355030    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355096 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355080    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzc29\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-kube-api-access-xzc29\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355124    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355251 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355181    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355301 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355248    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355348 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355304    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355343    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0486bcec-d668-4e37-a171-d1ef285293e9-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355383    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355507 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355418    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355507 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355453    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355507 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355483    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.355654 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355545    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrph2\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-kube-api-access-lrph2\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355654 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355594    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355654 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355605    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355674    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355809    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355913 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355853    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.355963 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.355951    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.356178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.356112    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.356286 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.356248    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.356347 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.356298    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0486bcec-d668-4e37-a171-d1ef285293e9-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.357763 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.357741    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0486bcec-d668-4e37-a171-d1ef285293e9-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.357916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.357900    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.358358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.358330    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.358626 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.358609    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.383022 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.382954    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.383188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.383129    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzc29\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-kube-api-access-xzc29\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.383765 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.383741    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrph2\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-kube-api-access-lrph2\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.384573 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.384549    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.537393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.537357    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:11.588686 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.588657    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:11.681186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.681109    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:11.871344 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:11.871297    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b4bcad_310b_4c0a_a691_d5319b2f01bd.slice/crio-6e0c2d46af8c7ddffa2a59df360d8b5130da68ba9aaf0eec0077ec2db8e7f065 WatchSource:0}: Error finding container 6e0c2d46af8c7ddffa2a59df360d8b5130da68ba9aaf0eec0077ec2db8e7f065: Status 404 returned error can't find the container with id 6e0c2d46af8c7ddffa2a59df360d8b5130da68ba9aaf0eec0077ec2db8e7f065
Oct 29 22:57:11.874322 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.874283    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:11.874439 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.874368    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:11.874439 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.874405    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:11.876497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.876469    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"]
Oct 29 22:57:11.916580 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.916550    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"]
Oct 29 22:57:11.918592 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:11.918553    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0486bcec_d668_4e37_a171_d1ef285293e9.slice/crio-e268be4c4468e3fc91e4a3558d859ea102521151a62bf3eda7e5ce7ac6204745 WatchSource:0}: Error finding container e268be4c4468e3fc91e4a3558d859ea102521151a62bf3eda7e5ce7ac6204745: Status 404 returned error can't find the container with id e268be4c4468e3fc91e4a3558d859ea102521151a62bf3eda7e5ce7ac6204745
Oct 29 22:57:11.921476 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.921442    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:11.921636 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.921520    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:11.921636 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.921572    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:11.982218 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.982181    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp" event={"ID":"219fa173-07df-42dc-9cb8-c5f5d2043fb0","Type":"ContainerStarted","Data":"6633b1d28c3bd8880e1e869b166782f5ded1b4274519be73387583bb988a53d8"}
Oct 29 22:57:11.982477 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.982453    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp" event={"ID":"219fa173-07df-42dc-9cb8-c5f5d2043fb0","Type":"ContainerStarted","Data":"165755a906d3f8ecd6682765c03c62a9b3e8e391dede47d11ec5712fe127c4fd"}
Oct 29 22:57:11.984600 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.984572    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" event={"ID":"0486bcec-d668-4e37-a171-d1ef285293e9","Type":"ContainerStarted","Data":"e268be4c4468e3fc91e4a3558d859ea102521151a62bf3eda7e5ce7ac6204745"}
Oct 29 22:57:11.990647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.990616    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" event={"ID":"89b4bcad-310b-4c0a-a691-d5319b2f01bd","Type":"ContainerStarted","Data":"8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d"}
Oct 29 22:57:11.990794 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:11.990684    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" event={"ID":"89b4bcad-310b-4c0a-a691-d5319b2f01bd","Type":"ContainerStarted","Data":"6e0c2d46af8c7ddffa2a59df360d8b5130da68ba9aaf0eec0077ec2db8e7f065"}
Oct 29 22:57:12.034676 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:12.032547    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp" podStartSLOduration=2.032528561 podStartE2EDuration="2.032528561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:12.030830789 +0000 UTC m=+889.133907063" watchObservedRunningTime="2026-10-29 22:57:12.032528561 +0000 UTC m=+889.135604834"
Oct 29 22:57:12.537776 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:12.537744    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:12.539257 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:12.539221    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:57:12.539397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:12.539277    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:57:13.025714 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.025662    4839 generic.go:356] "Generic (PLEG): container finished" podID="219fa173-07df-42dc-9cb8-c5f5d2043fb0" containerID="6633b1d28c3bd8880e1e869b166782f5ded1b4274519be73387583bb988a53d8" exitCode=0
Oct 29 22:57:13.032369 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.031591    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp" event={"ID":"219fa173-07df-42dc-9cb8-c5f5d2043fb0","Type":"ContainerDied","Data":"6633b1d28c3bd8880e1e869b166782f5ded1b4274519be73387583bb988a53d8"}
Oct 29 22:57:13.032369 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.031653    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" event={"ID":"0486bcec-d668-4e37-a171-d1ef285293e9","Type":"ContainerStarted","Data":"fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574"}
Oct 29 22:57:13.111229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.089098    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podStartSLOduration=2.089076016 podStartE2EDuration="2.089076016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:12.096898349 +0000 UTC m=+889.199974602" watchObservedRunningTime="2026-10-29 22:57:13.089076016 +0000 UTC m=+890.192152291"
Oct 29 22:57:13.142955 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.142900    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podStartSLOduration=2.142885254 podStartE2EDuration="2.142885254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:13.141174823 +0000 UTC m=+890.244251092" watchObservedRunningTime="2026-10-29 22:57:13.142885254 +0000 UTC m=+890.245961563"
Oct 29 22:57:13.446417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.446306    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:13.550231 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.543874    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:57:13.550231 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.544048    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:57:13.593173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.591226    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:13.593173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.592072    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:57:13.593173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:13.592140    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:57:14.362246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.362209    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:14.519171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.517526    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-xks-ca\") pod \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") "
Oct 29 22:57:14.519171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.517659    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219fa173-07df-42dc-9cb8-c5f5d2043fb0-kube-api-access-fjg2m\" (UniqueName: \"kubernetes.io/projected/219fa173-07df-42dc-9cb8-c5f5d2043fb0-kube-api-access-fjg2m\") pod \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") "
Oct 29 22:57:14.519171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.517909    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-openshift-ca\") pod \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\" (UID: \"219fa173-07df-42dc-9cb8-c5f5d2043fb0\") "
Oct 29 22:57:14.523201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.520398    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-openshift-ca" pod "219fa173-07df-42dc-9cb8-c5f5d2043fb0" (UID: "219fa173-07df-42dc-9cb8-c5f5d2043fb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:14.523630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.523599    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-xks-ca" pod "219fa173-07df-42dc-9cb8-c5f5d2043fb0" (UID: "219fa173-07df-42dc-9cb8-c5f5d2043fb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:14.524952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.524921    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219fa173-07df-42dc-9cb8-c5f5d2043fb0-kube-api-access-fjg2m" pod "219fa173-07df-42dc-9cb8-c5f5d2043fb0" (UID: "219fa173-07df-42dc-9cb8-c5f5d2043fb0"). InnerVolumeSpecName "kube-api-access-fjg2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:14.538338 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.538313    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:57:14.538447 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.538368    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:57:14.592226 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.592022    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:57:14.592226 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.592105    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:57:14.620052 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.620006    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:14.620052 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.620039    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219fa173-07df-42dc-9cb8-c5f5d2043fb0-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:14.620052 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:14.620054    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjg2m\" (UniqueName: \"kubernetes.io/projected/219fa173-07df-42dc-9cb8-c5f5d2043fb0-kube-api-access-fjg2m\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:15.044358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.044325    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp" event={"ID":"219fa173-07df-42dc-9cb8-c5f5d2043fb0","Type":"ContainerDied","Data":"165755a906d3f8ecd6682765c03c62a9b3e8e391dede47d11ec5712fe127c4fd"}
Oct 29 22:57:15.044358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.044350    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"
Oct 29 22:57:15.044511 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.044379    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="165755a906d3f8ecd6682765c03c62a9b3e8e391dede47d11ec5712fe127c4fd"
Oct 29 22:57:15.124178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.123902    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:15.268623 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.268589    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:15.537906 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.537877    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:57:15.538333 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.537928    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:57:15.592544 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.592507    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:57:15.594403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:15.594369    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:57:16.538047 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:16.538008    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:57:16.538484 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:16.538074    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:57:16.598606 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:16.598568    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:57:16.598939 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:16.598910    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:57:16.990137 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:16.990001    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:17.538526 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.538482    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:57:17.538919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.538535    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:57:17.539842 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.539821    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"]
Oct 29 22:57:17.540407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.540393    4839 cpu_manager.go:414] "containerMap: removing container" podUID="219fa173-07df-42dc-9cb8-c5f5d2043fb0" containerName="revoke-keys"
Oct 29 22:57:17.540441 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.540409    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219fa173-07df-42dc-9cb8-c5f5d2043fb0" containerName="revoke-keys"
Oct 29 22:57:17.540501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.540492    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="219fa173-07df-42dc-9cb8-c5f5d2043fb0" containerName="revoke-keys"
Oct 29 22:57:17.544209 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.544190    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.561641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.561615    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"]
Oct 29 22:57:17.582171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.577443    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:17.582171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.578174    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:17.618180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.590613    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:57:17.618180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.590671    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:57:17.626626 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.626325    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.626626 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.626408    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.626626 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.626527    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvngt\" (UniqueName: \"kubernetes.io/projected/983666a3-7eba-422d-b110-a8ad96351dc1-kube-api-access-vvngt\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.727574 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.727530    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.727751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.727599    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.727751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.727715    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvngt\" (UniqueName: \"kubernetes.io/projected/983666a3-7eba-422d-b110-a8ad96351dc1-kube-api-access-vvngt\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.728330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.728308    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.730177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.730136    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.748716 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.748688    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvngt\" (UniqueName: \"kubernetes.io/projected/983666a3-7eba-422d-b110-a8ad96351dc1-kube-api-access-vvngt\") pod \"maas-api-revoke-sub-fc1b5fe5-5vnx8\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:17.860598 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:17.860511    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:18.274046 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:18.274002    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983666a3_7eba_422d_b110_a8ad96351dc1.slice/crio-0ec30b4ed6bc2ee357d5ab41f420dca413eb5f3d647fb2415f1f40520024e9d3 WatchSource:0}: Error finding container 0ec30b4ed6bc2ee357d5ab41f420dca413eb5f3d647fb2415f1f40520024e9d3: Status 404 returned error can't find the container with id 0ec30b4ed6bc2ee357d5ab41f420dca413eb5f3d647fb2415f1f40520024e9d3
Oct 29 22:57:18.275764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:18.275740    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"]
Oct 29 22:57:18.538569 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:18.538488    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 29 22:57:18.538569 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:18.538546    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 29 22:57:18.589211 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:18.589185    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 29 22:57:18.589329 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:18.589228    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 29 22:57:18.713077 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:18.713042    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:19.120162 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.116714    4839 generic.go:356] "Generic (PLEG): container finished" podID="983666a3-7eba-422d-b110-a8ad96351dc1" containerID="cb9c4c8a61ba170360e0d51e9790fbb145e96edc0509078a197b29f1d3aced8c" exitCode=0
Oct 29 22:57:19.120162 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.116911    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8" event={"ID":"983666a3-7eba-422d-b110-a8ad96351dc1","Type":"ContainerDied","Data":"cb9c4c8a61ba170360e0d51e9790fbb145e96edc0509078a197b29f1d3aced8c"}
Oct 29 22:57:19.120162 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.116953    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8" event={"ID":"983666a3-7eba-422d-b110-a8ad96351dc1","Type":"ContainerStarted","Data":"0ec30b4ed6bc2ee357d5ab41f420dca413eb5f3d647fb2415f1f40520024e9d3"}
Oct 29 22:57:19.824953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.824332    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:19.825461 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.825336    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:19.826343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.826296    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 22:57:19.990186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.990137    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:19.990864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.990841    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:19.994424 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:19.994400    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 22:57:20.318960 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.318927    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:20.327964 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.327941    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:20.394220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.393848    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-xks-ca\") pod \"983666a3-7eba-422d-b110-a8ad96351dc1\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") "
Oct 29 22:57:20.394220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.393954    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983666a3-7eba-422d-b110-a8ad96351dc1-kube-api-access-vvngt\" (UniqueName: \"kubernetes.io/projected/983666a3-7eba-422d-b110-a8ad96351dc1-kube-api-access-vvngt\") pod \"983666a3-7eba-422d-b110-a8ad96351dc1\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") "
Oct 29 22:57:20.394220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.393990    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-openshift-ca\") pod \"983666a3-7eba-422d-b110-a8ad96351dc1\" (UID: \"983666a3-7eba-422d-b110-a8ad96351dc1\") "
Oct 29 22:57:20.412245 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.408790    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-openshift-ca" pod "983666a3-7eba-422d-b110-a8ad96351dc1" (UID: "983666a3-7eba-422d-b110-a8ad96351dc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:20.412245 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.409581    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:20.413356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.412689    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983666a3-7eba-422d-b110-a8ad96351dc1-kube-api-access-vvngt" pod "983666a3-7eba-422d-b110-a8ad96351dc1" (UID: "983666a3-7eba-422d-b110-a8ad96351dc1"). InnerVolumeSpecName "kube-api-access-vvngt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:20.428239 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.426771    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-xks-ca" pod "983666a3-7eba-422d-b110-a8ad96351dc1" (UID: "983666a3-7eba-422d-b110-a8ad96351dc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:20.511407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.511374    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvngt\" (UniqueName: \"kubernetes.io/projected/983666a3-7eba-422d-b110-a8ad96351dc1-kube-api-access-vvngt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:20.511407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:20.511404    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983666a3-7eba-422d-b110-a8ad96351dc1-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:21.009972 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.009937    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"]
Oct 29 22:57:21.010535 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.010505    4839 cpu_manager.go:414] "containerMap: removing container" podUID="983666a3-7eba-422d-b110-a8ad96351dc1" containerName="revoke-keys"
Oct 29 22:57:21.010535 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.010524    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983666a3-7eba-422d-b110-a8ad96351dc1" containerName="revoke-keys"
Oct 29 22:57:21.010756 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.010634    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="983666a3-7eba-422d-b110-a8ad96351dc1" containerName="revoke-keys"
Oct 29 22:57:21.013983 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.013965    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.036683 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.036659    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"]
Oct 29 22:57:21.118615 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.118582    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnjt6\" (UniqueName: \"kubernetes.io/projected/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-kube-api-access-hnjt6\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.118807 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.118773    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.118914 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.118886    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.156667 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.156635    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"
Oct 29 22:57:21.156857 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.156665    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8" event={"ID":"983666a3-7eba-422d-b110-a8ad96351dc1","Type":"ContainerDied","Data":"0ec30b4ed6bc2ee357d5ab41f420dca413eb5f3d647fb2415f1f40520024e9d3"}
Oct 29 22:57:21.156857 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.156719    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ec30b4ed6bc2ee357d5ab41f420dca413eb5f3d647fb2415f1f40520024e9d3"
Oct 29 22:57:21.220470 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.220434    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.220646 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.220501    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.220852 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.220816    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnjt6\" (UniqueName: \"kubernetes.io/projected/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-kube-api-access-hnjt6\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.221309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.221276    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.223382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.223358    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.243566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.243535    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnjt6\" (UniqueName: \"kubernetes.io/projected/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-kube-api-access-hnjt6\") pod \"maas-api-revoke-sub-574b2b00-gww2q\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.329496 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.329463    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:21.628969 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:21.628938    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff320260_9e76_4ee5_b8aa_9910bd6e3c25.slice/crio-6584fd466a92d3325a6569fea4871a928efb526f11fde4963ac5cf04f17bd918 WatchSource:0}: Error finding container 6584fd466a92d3325a6569fea4871a928efb526f11fde4963ac5cf04f17bd918: Status 404 returned error can't find the container with id 6584fd466a92d3325a6569fea4871a928efb526f11fde4963ac5cf04f17bd918
Oct 29 22:57:21.631881 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:21.631861    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"]
Oct 29 22:57:22.167072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:22.166986    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q" event={"ID":"ff320260-9e76-4ee5-b8aa-9910bd6e3c25","Type":"ContainerStarted","Data":"3304443c26b53c5fadd661ee5c285e2e7e311f8aeff5c714f9346bbdf08c3b1a"}
Oct 29 22:57:22.167072 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:22.167049    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q" event={"ID":"ff320260-9e76-4ee5-b8aa-9910bd6e3c25","Type":"ContainerStarted","Data":"6584fd466a92d3325a6569fea4871a928efb526f11fde4963ac5cf04f17bd918"}
Oct 29 22:57:22.200562 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:22.200527    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:22.210552 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:22.210504    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q" podStartSLOduration=2.210491631 podStartE2EDuration="2.210491631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:22.209010581 +0000 UTC m=+899.312086852" watchObservedRunningTime="2026-10-29 22:57:22.210491631 +0000 UTC m=+899.313567902"
Oct 29 22:57:23.174823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:23.174786    4839 generic.go:356] "Generic (PLEG): container finished" podID="ff320260-9e76-4ee5-b8aa-9910bd6e3c25" containerID="3304443c26b53c5fadd661ee5c285e2e7e311f8aeff5c714f9346bbdf08c3b1a" exitCode=0
Oct 29 22:57:23.175263 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:23.174870    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q" event={"ID":"ff320260-9e76-4ee5-b8aa-9910bd6e3c25","Type":"ContainerDied","Data":"3304443c26b53c5fadd661ee5c285e2e7e311f8aeff5c714f9346bbdf08c3b1a"}
Oct 29 22:57:23.726519 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:23.726479    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:24.351732 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.351704    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:24.473256 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.473176    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-openshift-ca\") pod \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") "
Oct 29 22:57:24.473411 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.473383    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-kube-api-access-hnjt6\" (UniqueName: \"kubernetes.io/projected/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-kube-api-access-hnjt6\") pod \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") "
Oct 29 22:57:24.473411 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.473406    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-xks-ca\") pod \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\" (UID: \"ff320260-9e76-4ee5-b8aa-9910bd6e3c25\") "
Oct 29 22:57:24.473541 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.473518    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-openshift-ca" pod "ff320260-9e76-4ee5-b8aa-9910bd6e3c25" (UID: "ff320260-9e76-4ee5-b8aa-9910bd6e3c25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:24.473817 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.473801    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:24.475531 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.475505    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-xks-ca" pod "ff320260-9e76-4ee5-b8aa-9910bd6e3c25" (UID: "ff320260-9e76-4ee5-b8aa-9910bd6e3c25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:24.475634 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.475597    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-kube-api-access-hnjt6" pod "ff320260-9e76-4ee5-b8aa-9910bd6e3c25" (UID: "ff320260-9e76-4ee5-b8aa-9910bd6e3c25"). InnerVolumeSpecName "kube-api-access-hnjt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:24.575125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.575088    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnjt6\" (UniqueName: \"kubernetes.io/projected/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-kube-api-access-hnjt6\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:24.575125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:24.575121    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff320260-9e76-4ee5-b8aa-9910bd6e3c25-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:25.190577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.190535    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q" event={"ID":"ff320260-9e76-4ee5-b8aa-9910bd6e3c25","Type":"ContainerDied","Data":"6584fd466a92d3325a6569fea4871a928efb526f11fde4963ac5cf04f17bd918"}
Oct 29 22:57:25.190577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.190556    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"
Oct 29 22:57:25.190836 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.190590    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6584fd466a92d3325a6569fea4871a928efb526f11fde4963ac5cf04f17bd918"
Oct 29 22:57:25.389853 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.389773    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:25.797195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.797157    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"]
Oct 29 22:57:25.797752 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.797737    4839 cpu_manager.go:414] "containerMap: removing container" podUID="ff320260-9e76-4ee5-b8aa-9910bd6e3c25" containerName="revoke-keys"
Oct 29 22:57:25.797798 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.797753    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff320260-9e76-4ee5-b8aa-9910bd6e3c25" containerName="revoke-keys"
Oct 29 22:57:25.797838 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.797834    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff320260-9e76-4ee5-b8aa-9910bd6e3c25" containerName="revoke-keys"
Oct 29 22:57:25.801481 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.801460    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.839660 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.839613    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"]
Oct 29 22:57:25.901423 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901384    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901423 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901421    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901629 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901444    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j984\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-kube-api-access-6j984\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901629 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901562    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f4df42-81b7-4660-82b4-04c567edc874-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901692    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901738 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901717    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901774 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901747    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901914 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901897    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:25.901956 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:25.901945    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f4df42-81b7-4660-82b4-04c567edc874-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.002716 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.002681    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f4df42-81b7-4660-82b4-04c567edc874-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.002896 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.002773    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.002896 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.002797    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.002896 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.002828    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.002902    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.002933    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f4df42-81b7-4660-82b4-04c567edc874-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.002981    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.003006    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.003034    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j984\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-kube-api-access-6j984\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003384 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.003262    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-certs\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003384 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.003318    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.003410    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-credential-socket\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.003467    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-data\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.003762 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.003741    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f4df42-81b7-4660-82b4-04c567edc874-istiod-ca-cert\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.005208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.005185    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-envoy\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.005208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.005206    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f4df42-81b7-4660-82b4-04c567edc874-istio-podinfo\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.024135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.024107    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-istio-token\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.024416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.024395    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j984\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-kube-api-access-6j984\") pod \"e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") " pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.120096 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.120020    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:26.358181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.358127    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"]
Oct 29 22:57:26.364438 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.364407    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.399470 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.398796    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"]
Oct 29 22:57:26.459063 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:26.459025    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f4df42_81b7_4660_82b4_04c567edc874.slice/crio-73a13c829657a416d8b7ca683eb42669e07981d25789b540866047087e953831 WatchSource:0}: Error finding container 73a13c829657a416d8b7ca683eb42669e07981d25789b540866047087e953831: Status 404 returned error can't find the container with id 73a13c829657a416d8b7ca683eb42669e07981d25789b540866047087e953831
Oct 29 22:57:26.460178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.460128    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"]
Oct 29 22:57:26.466250 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.466214    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:26.466344 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.466287    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:26.466344 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.466317    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:26.510631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.510600    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.510753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.510730    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.510796 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.510768    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.510848 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.510799    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.510965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.510937    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.511047 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.511031    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.511159 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.511129    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.511221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.511203    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.511340 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.511322    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pn82\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-kube-api-access-6pn82\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.612963 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.612927    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.612977    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613006    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613037    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613070    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613107    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613143    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613518 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613221    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pn82\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-kube-api-access-6pn82\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613518 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613302    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.613911 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.613882    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.614112 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.614084    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.614373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.614332    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.614373 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.614347    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.614527 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.614435    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.616336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.616308    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.616505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.616485    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.634833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.634764    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pn82\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-kube-api-access-6pn82\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.635239 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.635213    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:26.680889 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:26.680855    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:27.016248 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.016201    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:27.016387 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.016296    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:27.016387 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.016335    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:27.019804 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.019779    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"]
Oct 29 22:57:27.216491 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.216450    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" event={"ID":"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f","Type":"ContainerStarted","Data":"0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90"}
Oct 29 22:57:27.216685 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.216522    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" event={"ID":"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f","Type":"ContainerStarted","Data":"ef7b9208c4a83bf01c434fafb263570da30551a55b97392703fd45728dd4de6e"}
Oct 29 22:57:27.218296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.218271    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" event={"ID":"10f4df42-81b7-4660-82b4-04c567edc874","Type":"ContainerStarted","Data":"62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66"}
Oct 29 22:57:27.218421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.218312    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" event={"ID":"10f4df42-81b7-4660-82b4-04c567edc874","Type":"ContainerStarted","Data":"73a13c829657a416d8b7ca683eb42669e07981d25789b540866047087e953831"}
Oct 29 22:57:27.244500 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.244467    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:27.263761 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.263647    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podStartSLOduration=1.263632504 podStartE2EDuration="1.263632504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:27.26100613 +0000 UTC m=+904.364082400" watchObservedRunningTime="2026-10-29 22:57:27.263632504 +0000 UTC m=+904.366708775"
Oct 29 22:57:27.346557 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.346462    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podStartSLOduration=2.346425791 podStartE2EDuration="2.346425791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:27.318842199 +0000 UTC m=+904.421918510" watchObservedRunningTime="2026-10-29 22:57:27.346425791 +0000 UTC m=+904.449502066"
Oct 29 22:57:27.688174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:27.684269    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:28.122171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:28.121370    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:28.125166 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:28.124267    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:28.125166 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:28.124343    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:28.687172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:28.684379    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:28.687172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:28.684464    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:29.142257 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:29.142210    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:29.142774 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:29.142277    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:29.717166 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:29.716279    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:29.717166 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:29.716368    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:29.778170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:29.772583    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:29.980236 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:29.977163    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.124172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.123535    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:30.124172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.123598    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:30.140349 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.140240    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.156170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.156100    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.156832 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.156803    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.566243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.557244    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.566243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.558023    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.682172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.681345    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:30.682172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.681409    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:30.847230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.846244    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.851170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.847560    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.961171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.956241    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:30.961171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.956643    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.988686 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.988634    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:30.989460 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:30.989432    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:57:31.194235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.154424    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.194235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.157032    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:31.194235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.157487    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:57:31.194235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.184800    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:31.194235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.184855    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:31.565212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.558253    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:31.565212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.560216    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.565212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.560922    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:57:31.565212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.560985    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:57:31.703755 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.703282    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:31.703755 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.703348    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:31.859765 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.859668    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:31.874172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.861821    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.891170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.888106    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:57:31.891170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.888209    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:57:31.985212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.960730    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:31.985212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.960760    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:57:31.985212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.960912    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:57:31.985212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.960968    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:57:31.996845 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:31.996803    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.179237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.164235    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.179237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.164674    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.179237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.166548    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:57:32.232175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.213960    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:32.236173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.232799    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:32.362346 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.362298    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:32.362650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.362599    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.559172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.534489    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 29 22:57:32.602172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.559973    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.670304 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.587952    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.698197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.694025    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:32.698197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.694088    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:32.866844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.866027    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.881687 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.881260    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:57:32.930170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.908169    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded"
Oct 29 22:57:32.973287 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.973229    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:32.974778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:32.974734    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.010313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.010266    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.234614 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.234519    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.262171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.235997    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.262505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.262465    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:33.263061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.253909    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.664180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.639655    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.695419 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.672481    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.696330 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.683293    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:33.696564 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.696537    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:33.815572 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:57:33.815488    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:57:33.827713 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.827667    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.879750 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.879705    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:57:33.936228 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.896131    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:33.979306 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.979255    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.003174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:33.979901    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.017293 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.014771    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.314439 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.314386    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.315087 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.315033    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:34.344653 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.344609    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.361195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.359730    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.721479 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.710621    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:34.823173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.745828    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:34.823173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.791211    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:34.823173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.745871    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:34.901338 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.888401    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.121864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.981641    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.121864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:34.981777    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.121864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.008478    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.121864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.022422    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.200173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.196856    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:35.200470 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.200434    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:35.222173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.215418    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:35.380220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.376316    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.381616 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.380832    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.779231 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.778106    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 29 22:57:35.779231 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.778491    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:35.779231 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.778539    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:35.780524 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.780484    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:35.780746 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.780709    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.795173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.792563    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:35.795173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.793448    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:35.895180 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.891319    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:35.928171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:35.903230    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.049899 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.049796    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.066664 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.066622    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:57:36.068037 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.068006    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:57:36.069315 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.069275    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.069981 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.069958    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:36.082722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.082679    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.126174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.111893    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"]
Oct 29 22:57:36.126551 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.126519    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.126757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.126732    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:36.211364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.206230    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:36.211364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.206291    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.277294 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.277244    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:36.335708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.333511    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.382200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.381819    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.472280 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.433588    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:57:36.506475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.506438    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"]
Oct 29 22:57:36.506824 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.506804    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.532199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.531258    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-trqpq" podUID="ef38dee7-98b1-4e20-992d-240005ef83e1" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.12:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.551525 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.551496    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:36.552397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.552371    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:36.593224 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.553246    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:57:36.725972 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.694323    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:36.726269 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.726235    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:36.744270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.725290    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.769168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.767295    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.769168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.767435    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.769168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.767485    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxwz5\" (UniqueName: \"kubernetes.io/projected/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-kube-api-access-jxwz5\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.798174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.794992    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.821181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.800428    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:36.890227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.882852    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.890227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.882938    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.890227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.882981    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxwz5\" (UniqueName: \"kubernetes.io/projected/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-kube-api-access-jxwz5\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.905705 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.905671    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.964498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.963037    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxwz5\" (UniqueName: \"kubernetes.io/projected/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-kube-api-access-jxwz5\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:36.967100 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:36.964933    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:57:37.042175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.039547    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b780ad0-nzfv6\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:37.076987 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.076546    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:37.130571 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.130285    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.130571 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.130342    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:37.145790 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.145706    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:37.153177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.150502    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:37.159258 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.159234    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:57:37.159452 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.159440    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:57:37.159610 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.159581    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:57:37.159676 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.159650    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:57:37.159734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.159709    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:57:37.165239 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.165221    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:57:37.167894 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.167093    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:57:37.230220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.222221    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:37.624195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.604248    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:37.630745 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:37.630709    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da4be8e_44a6_4ab1_a36c_6f4a2d1e7205.slice/crio-55b1495a5396eaa9349e9a523ed40c6c4e0218c1d1e3a5c874af363973142d46 WatchSource:0}: Error finding container 55b1495a5396eaa9349e9a523ed40c6c4e0218c1d1e3a5c874af363973142d46: Status 404 returned error can't find the container with id 55b1495a5396eaa9349e9a523ed40c6c4e0218c1d1e3a5c874af363973142d46
Oct 29 22:57:37.634482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.634454    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"]
Oct 29 22:57:37.720172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.708302    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:37.720172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:37.708365    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:38.121139 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:38.121105    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 29 22:57:38.121319 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:38.121209    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 29 22:57:38.645986 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:38.645946    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6" event={"ID":"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205","Type":"ContainerStarted","Data":"911a2d0c3197f7563c3814a60323cb3537955235f02a80563eebfe1a06d46e1d"}
Oct 29 22:57:38.646396 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:38.646006    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6" event={"ID":"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205","Type":"ContainerStarted","Data":"55b1495a5396eaa9349e9a523ed40c6c4e0218c1d1e3a5c874af363973142d46"}
Oct 29 22:57:38.681316 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:38.681270    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Oct 29 22:57:38.681497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:38.681338    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Oct 29 22:57:39.462661 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:39.462624    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:39.657686 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:39.657650    4839 generic.go:356] "Generic (PLEG): container finished" podID="2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" containerID="911a2d0c3197f7563c3814a60323cb3537955235f02a80563eebfe1a06d46e1d" exitCode=0
Oct 29 22:57:39.658176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:39.657801    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6" event={"ID":"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205","Type":"ContainerDied","Data":"911a2d0c3197f7563c3814a60323cb3537955235f02a80563eebfe1a06d46e1d"}
Oct 29 22:57:40.122119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:40.122076    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:40.122343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:40.122137    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.685199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:40.682657    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:40.685199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:40.682729    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:40.949734 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:40.949661    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:41.120939 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.120903    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:41.123346 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.123289    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:57:41.123519 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.123394    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:41.142823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.142796    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-kube-api-access-jxwz5\" (UniqueName: \"kubernetes.io/projected/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-kube-api-access-jxwz5\") pod \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") "
Oct 29 22:57:41.142999 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.142946    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-openshift-ca\") pod \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") "
Oct 29 22:57:41.142999 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.142977    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-xks-ca\") pod \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\" (UID: \"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205\") "
Oct 29 22:57:41.143323 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.143291    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-openshift-ca" pod "2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" (UID: "2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:41.143607 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.143589    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:41.145119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.145089    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-kube-api-access-jxwz5" pod "2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" (UID: "2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205"). InnerVolumeSpecName "kube-api-access-jxwz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:41.145417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.145396    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-xks-ca" pod "2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" (UID: "2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:41.183481 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.183042    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:41.244887 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.244788    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxwz5\" (UniqueName: \"kubernetes.io/projected/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-kube-api-access-jxwz5\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:41.244887 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.244823    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:41.413620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.413582    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:41.687677 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.687636    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 22:57:41.688260 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.687736    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 22:57:41.689015 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.688991    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:41.697019 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.696995    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"
Oct 29 22:57:41.697131 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.697018    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6" event={"ID":"2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205","Type":"ContainerDied","Data":"55b1495a5396eaa9349e9a523ed40c6c4e0218c1d1e3a5c874af363973142d46"}
Oct 29 22:57:41.697131 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.697065    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55b1495a5396eaa9349e9a523ed40c6c4e0218c1d1e3a5c874af363973142d46"
Oct 29 22:57:41.698195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.698173    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 22:57:41.739309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:41.739279    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:42.711949 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:42.711923    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 22:57:43.575911 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:43.575566    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:45.583761 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:45.583723    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:47.340963 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.340929    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"]
Oct 29 22:57:47.341531 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.341514    4839 cpu_manager.go:414] "containerMap: removing container" podUID="2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" containerName="revoke-keys"
Oct 29 22:57:47.341573 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.341532    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" containerName="revoke-keys"
Oct 29 22:57:47.341640 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.341631    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" containerName="revoke-keys"
Oct 29 22:57:47.346594 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.346575    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.381108 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.381078    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:47.381108 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.381098    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:47.385499 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.385474    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"]
Oct 29 22:57:47.437212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.437175    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8765bc57-d592-4213-a88c-e5462620b637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.437390 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.437233    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8765bc57-d592-4213-a88c-e5462620b637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.437390 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.437294    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82g7\" (UniqueName: \"kubernetes.io/projected/8765bc57-d592-4213-a88c-e5462620b637-kube-api-access-b82g7\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.539433 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.539390    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8765bc57-d592-4213-a88c-e5462620b637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.542809 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.542767    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8765bc57-d592-4213-a88c-e5462620b637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.543102 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.543078    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82g7\" (UniqueName: \"kubernetes.io/projected/8765bc57-d592-4213-a88c-e5462620b637-kube-api-access-b82g7\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.570192 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.544541    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8765bc57-d592-4213-a88c-e5462620b637-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.603876 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.566669    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8765bc57-d592-4213-a88c-e5462620b637-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.603876 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.588045    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82g7\" (UniqueName: \"kubernetes.io/projected/8765bc57-d592-4213-a88c-e5462620b637-kube-api-access-b82g7\") pod \"maas-api-revoke-sub-6b4209b7-8np2h\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.636182 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.636107    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:47.660023 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.659992    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:47.969096 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:47.969060    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8765bc57_d592_4213_a88c_e5462620b637.slice/crio-2b84c4b102c34c6ca69ba5b96b6b44f0496cbacd2a7994716e0fe3a4a30fa0c3 WatchSource:0}: Error finding container 2b84c4b102c34c6ca69ba5b96b6b44f0496cbacd2a7994716e0fe3a4a30fa0c3: Status 404 returned error can't find the container with id 2b84c4b102c34c6ca69ba5b96b6b44f0496cbacd2a7994716e0fe3a4a30fa0c3
Oct 29 22:57:47.971639 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:47.971613    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"]
Oct 29 22:57:48.757916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:48.757831    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h" event={"ID":"8765bc57-d592-4213-a88c-e5462620b637","Type":"ContainerStarted","Data":"7cd9e4d8b6a078056672b610d2a2de5c584a7698bc5c530c2174290ef1c43e2f"}
Oct 29 22:57:48.757916 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:48.757880    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h" event={"ID":"8765bc57-d592-4213-a88c-e5462620b637","Type":"ContainerStarted","Data":"2b84c4b102c34c6ca69ba5b96b6b44f0496cbacd2a7994716e0fe3a4a30fa0c3"}
Oct 29 22:57:48.804426 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:48.804366    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h" podStartSLOduration=1.804348578 podStartE2EDuration="1.804348578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:48.802193908 +0000 UTC m=+925.905270181" watchObservedRunningTime="2026-10-29 22:57:48.804348578 +0000 UTC m=+925.907424852"
Oct 29 22:57:49.478170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:49.470602    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:49.768064 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:49.768027    4839 generic.go:356] "Generic (PLEG): container finished" podID="8765bc57-d592-4213-a88c-e5462620b637" containerID="7cd9e4d8b6a078056672b610d2a2de5c584a7698bc5c530c2174290ef1c43e2f" exitCode=0
Oct 29 22:57:49.768450 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:49.768067    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h" event={"ID":"8765bc57-d592-4213-a88c-e5462620b637","Type":"ContainerDied","Data":"7cd9e4d8b6a078056672b610d2a2de5c584a7698bc5c530c2174290ef1c43e2f"}
Oct 29 22:57:50.953197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:50.953166    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:50.988415 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:50.988381    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8765bc57-d592-4213-a88c-e5462620b637-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8765bc57-d592-4213-a88c-e5462620b637-maas-api-xks-ca\") pod \"8765bc57-d592-4213-a88c-e5462620b637\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") "
Oct 29 22:57:50.988575 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:50.988563    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8765bc57-d592-4213-a88c-e5462620b637-kube-api-access-b82g7\" (UniqueName: \"kubernetes.io/projected/8765bc57-d592-4213-a88c-e5462620b637-kube-api-access-b82g7\") pod \"8765bc57-d592-4213-a88c-e5462620b637\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") "
Oct 29 22:57:50.988618 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:50.988588    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8765bc57-d592-4213-a88c-e5462620b637-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8765bc57-d592-4213-a88c-e5462620b637-maas-api-openshift-ca\") pod \"8765bc57-d592-4213-a88c-e5462620b637\" (UID: \"8765bc57-d592-4213-a88c-e5462620b637\") "
Oct 29 22:57:50.988988 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:50.988958    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8765bc57-d592-4213-a88c-e5462620b637-maas-api-openshift-ca" pod "8765bc57-d592-4213-a88c-e5462620b637" (UID: "8765bc57-d592-4213-a88c-e5462620b637"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:50.990498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:50.990475    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8765bc57-d592-4213-a88c-e5462620b637-maas-api-xks-ca" pod "8765bc57-d592-4213-a88c-e5462620b637" (UID: "8765bc57-d592-4213-a88c-e5462620b637"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:50.990768 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:50.990748    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8765bc57-d592-4213-a88c-e5462620b637-kube-api-access-b82g7" pod "8765bc57-d592-4213-a88c-e5462620b637" (UID: "8765bc57-d592-4213-a88c-e5462620b637"). InnerVolumeSpecName "kube-api-access-b82g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:51.090919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:51.090832    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82g7\" (UniqueName: \"kubernetes.io/projected/8765bc57-d592-4213-a88c-e5462620b637-kube-api-access-b82g7\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.090919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:51.090869    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8765bc57-d592-4213-a88c-e5462620b637-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.090919 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:51.090883    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8765bc57-d592-4213-a88c-e5462620b637-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:51.210397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:51.210356    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:51.786364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:51.786335    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"
Oct 29 22:57:51.786537 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:51.786331    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h" event={"ID":"8765bc57-d592-4213-a88c-e5462620b637","Type":"ContainerDied","Data":"2b84c4b102c34c6ca69ba5b96b6b44f0496cbacd2a7994716e0fe3a4a30fa0c3"}
Oct 29 22:57:51.786537 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:51.786492    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b84c4b102c34c6ca69ba5b96b6b44f0496cbacd2a7994716e0fe3a4a30fa0c3"
Oct 29 22:57:52.860767 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:52.860732    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:53.581806 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.581767    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"]
Oct 29 22:57:53.582350 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.582335    4839 cpu_manager.go:414] "containerMap: removing container" podUID="8765bc57-d592-4213-a88c-e5462620b637" containerName="revoke-keys"
Oct 29 22:57:53.582397 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.582351    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8765bc57-d592-4213-a88c-e5462620b637" containerName="revoke-keys"
Oct 29 22:57:53.582485 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.582476    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="8765bc57-d592-4213-a88c-e5462620b637" containerName="revoke-keys"
Oct 29 22:57:53.586227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.586206    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.606643 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.606618    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"]
Oct 29 22:57:53.615844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.615347    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:57:53.615844 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.615358    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:57:53.634939 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.634908    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.635091 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.634961    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8rl6\" (UniqueName: \"kubernetes.io/projected/198ed509-6ae3-4382-b9b1-16065f113c0c-kube-api-access-d8rl6\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.635247 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.635226    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.736130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.736090    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.736324 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.736207    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.736324 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.736255    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8rl6\" (UniqueName: \"kubernetes.io/projected/198ed509-6ae3-4382-b9b1-16065f113c0c-kube-api-access-d8rl6\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.737030 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.737003    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.738887 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.738861    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.761803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.761776    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8rl6\" (UniqueName: \"kubernetes.io/projected/198ed509-6ae3-4382-b9b1-16065f113c0c-kube-api-access-d8rl6\") pod \"maas-api-revoke-sub-d0c6ee35-xw59j\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:53.899971 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:53.899890    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:54.204847 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:57:54.204810    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198ed509_6ae3_4382_b9b1_16065f113c0c.slice/crio-2c4d090c981481a6b0ceeee5122b266d7dadd0a9a1a1719b2d81a81c614c519c WatchSource:0}: Error finding container 2c4d090c981481a6b0ceeee5122b266d7dadd0a9a1a1719b2d81a81c614c519c: Status 404 returned error can't find the container with id 2c4d090c981481a6b0ceeee5122b266d7dadd0a9a1a1719b2d81a81c614c519c
Oct 29 22:57:54.205932 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:54.205906    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"]
Oct 29 22:57:54.759364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:54.759078    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:54.869378 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:54.869338    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j" event={"ID":"198ed509-6ae3-4382-b9b1-16065f113c0c","Type":"ContainerStarted","Data":"087262b2d8e3ddca7eb4525b89c8363d543347de50b922e06e1128a773c58585"}
Oct 29 22:57:54.869707 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:54.869429    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j" event={"ID":"198ed509-6ae3-4382-b9b1-16065f113c0c","Type":"ContainerStarted","Data":"2c4d090c981481a6b0ceeee5122b266d7dadd0a9a1a1719b2d81a81c614c519c"}
Oct 29 22:57:54.926602 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:54.926548    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j" podStartSLOduration=1.926534841 podStartE2EDuration="1.926534841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:57:54.912894195 +0000 UTC m=+932.015970467" watchObservedRunningTime="2026-10-29 22:57:54.926534841 +0000 UTC m=+932.029611111"
Oct 29 22:57:55.877965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:55.877917    4839 generic.go:356] "Generic (PLEG): container finished" podID="198ed509-6ae3-4382-b9b1-16065f113c0c" containerID="087262b2d8e3ddca7eb4525b89c8363d543347de50b922e06e1128a773c58585" exitCode=0
Oct 29 22:57:55.878271 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:55.878000    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j" event={"ID":"198ed509-6ae3-4382-b9b1-16065f113c0c","Type":"ContainerDied","Data":"087262b2d8e3ddca7eb4525b89c8363d543347de50b922e06e1128a773c58585"}
Oct 29 22:57:56.266324 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.266285    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:57:56.425110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.425058    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"]
Oct 29 22:57:56.500763 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.500721    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"]
Oct 29 22:57:56.504513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.504474    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.510794 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.510762    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"]
Oct 29 22:57:56.515760 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.515727    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"]
Oct 29 22:57:56.515760 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.510917    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.582697 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.582657    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.582697 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.582707    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687941a-088e-4354-9d55-98861c7f308a-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.582953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.582731    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.582953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.582822    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.582953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.582854    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrqm\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-kube-api-access-kkrqm\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.582953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.582907    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.583111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583019    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.583111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583054    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.583111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583079    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.583111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583096    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.583303 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583278    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.583356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583303    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49wl\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-kube-api-access-h49wl\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.583407 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583392    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.583447 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583433    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.583497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583449    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687941a-088e-4354-9d55-98861c7f308a-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.583642 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583606    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.583720 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583659    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d4a2f-4950-471a-8de9-bba72662c7b8-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.583760 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.583741    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.684873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.684835    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.684873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.684884    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687941a-088e-4354-9d55-98861c7f308a-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685107 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.684997    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685107 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685022    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d4a2f-4950-471a-8de9-bba72662c7b8-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685107 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685043    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685276 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685105    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685276 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685131    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687941a-088e-4354-9d55-98861c7f308a-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685276 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685254    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685301    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685331    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrqm\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-kube-api-access-kkrqm\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685354    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685397    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685633 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685508    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-credential-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685633 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685531    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685740 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685636    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685740 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685685    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685740 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685716    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.685892 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685770    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685892 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685800    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49wl\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-kube-api-access-h49wl\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.685892 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685803    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-credential-socket\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.686079 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685932    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.686079 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.685965    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-certs\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.686301 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.686104    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687941a-088e-4354-9d55-98861c7f308a-istiod-ca-cert\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.686301 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.686281    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d4a2f-4950-471a-8de9-bba72662c7b8-istiod-ca-cert\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.686449 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.686348    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-certs\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.686449 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.686431    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-socket\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.686702 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.686675    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-data\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.686842 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.686818    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-data\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.688213 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.688184    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687941a-088e-4354-9d55-98861c7f308a-istio-podinfo\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.688416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.688385    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-podinfo\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.688842 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.688821    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-envoy\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.689187 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.689166    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-envoy\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.717391 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.717353    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-istio-token\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.717737 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.717691    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49wl\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-kube-api-access-h49wl\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.717884 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.717764    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrqm\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-kube-api-access-kkrqm\") pod \"e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") " pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.719730 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.719700    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-token\") pod \"e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") " pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:56.829386 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.829279    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:57:56.840011 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:56.839976    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:57:57.730172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:57.725855    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"]
Oct 29 22:57:57.733008 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:57.732950    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:57.733306 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:57.733269    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:57.733467 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:57.733446    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:58.063265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.029325    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"]
Oct 29 22:57:58.063265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.059273    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.123234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.098357    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:58.123234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.117279    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:58.123234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.117374    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:57:58.207350 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.191599    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.703549 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.700539    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.706086 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.705993    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" event={"ID":"a28d4a2f-4950-471a-8de9-bba72662c7b8","Type":"ContainerStarted","Data":"e62612923a661effbdefb17460f80db0b17208372a949e088a2a68220299467d"}
Oct 29 22:57:58.745786 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.745544    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" event={"ID":"d687941a-088e-4354-9d55-98861c7f308a","Type":"ContainerStarted","Data":"0af8eb18721c48b871b49af7b634194381287497d2878f433c01a5a1d39926af"}
Oct 29 22:57:58.773744 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.773352    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:57:58.836179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.830220    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.941911 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.876288    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.941911 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.890792    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.942561 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.942135    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-xks-ca\") pod \"198ed509-6ae3-4382-b9b1-16065f113c0c\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") "
Oct 29 22:57:58.993649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.978500    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:58.993649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.978884    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:57:58.993649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.980285    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-openshift-ca\") pod \"198ed509-6ae3-4382-b9b1-16065f113c0c\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") "
Oct 29 22:57:58.993649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:58.980396    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198ed509-6ae3-4382-b9b1-16065f113c0c-kube-api-access-d8rl6\" (UniqueName: \"kubernetes.io/projected/198ed509-6ae3-4382-b9b1-16065f113c0c-kube-api-access-d8rl6\") pod \"198ed509-6ae3-4382-b9b1-16065f113c0c\" (UID: \"198ed509-6ae3-4382-b9b1-16065f113c0c\") "
Oct 29 22:57:59.021057 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.017599    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-openshift-ca" pod "198ed509-6ae3-4382-b9b1-16065f113c0c" (UID: "198ed509-6ae3-4382-b9b1-16065f113c0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:57:59.084222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.083633    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:59.084222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.084166    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.198763 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.196524    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.251617 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.251564    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-xks-ca" pod "198ed509-6ae3-4382-b9b1-16065f113c0c" (UID: "198ed509-6ae3-4382-b9b1-16065f113c0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:57:59.280778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.274825    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:57:59.280778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.274961    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:57:59.343264 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.304758    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.343264 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.306778    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/198ed509-6ae3-4382-b9b1-16065f113c0c-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:59.567302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.547333    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198ed509-6ae3-4382-b9b1-16065f113c0c-kube-api-access-d8rl6" pod "198ed509-6ae3-4382-b9b1-16065f113c0c" (UID: "198ed509-6ae3-4382-b9b1-16065f113c0c"). InnerVolumeSpecName "kube-api-access-d8rl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:57:59.802083 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.568445    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8rl6\" (UniqueName: \"kubernetes.io/projected/198ed509-6ae3-4382-b9b1-16065f113c0c-kube-api-access-d8rl6\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:57:59.843169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.833465    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.843169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.833914    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:57:59.866192 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.862034    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:57:59.866192 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.865274    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:57:59.891173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.883486    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.891173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.883974    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:57:59.916176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.907506    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:57:59.916176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:57:59.907911    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:58:00.082272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.002362    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.082272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.002732    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:58:00.082272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.003349    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:58:00.082272 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.003403    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:58:00.104299 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.104252    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.148202 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.141683    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" event={"ID":"a28d4a2f-4950-471a-8de9-bba72662c7b8","Type":"ContainerStarted","Data":"25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5"}
Oct 29 22:58:00.148202 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.144287    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.206302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.186116    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.263173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.251409    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.320190 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.304190    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j" event={"ID":"198ed509-6ae3-4382-b9b1-16065f113c0c","Type":"ContainerDied","Data":"2c4d090c981481a6b0ceeee5122b266d7dadd0a9a1a1719b2d81a81c614c519c"}
Oct 29 22:58:00.320190 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.304260    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c4d090c981481a6b0ceeee5122b266d7dadd0a9a1a1719b2d81a81c614c519c"
Oct 29 22:58:00.320190 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.304342    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"
Oct 29 22:58:00.320190 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.317521    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.320190 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.317987    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:58:00.380270 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.379971    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" event={"ID":"d687941a-088e-4354-9d55-98861c7f308a","Type":"ContainerStarted","Data":"3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc"}
Oct 29 22:58:00.906492 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.906452    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:58:00.908286 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.908243    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:00.923177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.908495    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:01.646125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.920516    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:01.882589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.882485    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:01.902669 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.969710    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.094168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.007281    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.131197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.007329    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.152122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.238592    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.152122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.252431    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.152122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.358121    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.152122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:00.942113    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.152122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.238303    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.152122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.149537    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:02.152122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.055136    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:58:02.152747 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:01.937863    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.152908 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.152875    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:02.153062 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.118383    4839 patch_prober.go:29] interesting pod/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:02.153218 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.153194    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.170177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.162462    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.170177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.162537    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:02.257187 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:58:02.240901    4839 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.227s"
Oct 29 22:58:02.398459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.398420    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:58:02.437458 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.416825    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podStartSLOduration=6.416801759 podStartE2EDuration="6.416801759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:00.559255035 +0000 UTC m=+937.662331326" watchObservedRunningTime="2026-10-29 22:58:02.416801759 +0000 UTC m=+939.519878032"
Oct 29 22:58:02.616223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.574586    4839 patch_prober.go:29] interesting pod/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:02.616223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.574658    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:02.616223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.574956    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:02.847179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.838544    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.868178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.855242    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:02.871242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.845408    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:02.961327 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:02.949242    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:03.069312 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.069250    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.090191 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.069627    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": context deadline exceeded"
Oct 29 22:58:03.090888 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.069866    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.117233 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.098214    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.148075 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.147688    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.154381 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.154242    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.177185 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.147870    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.177185 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.147925    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.177185 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.147625    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.177185 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.147959    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:03.177185 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.147842    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:58:03.402237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.394273    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podStartSLOduration=7.394243653 podStartE2EDuration="7.394243653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:00.78070701 +0000 UTC m=+937.883783282" watchObservedRunningTime="2026-10-29 22:58:03.394243653 +0000 UTC m=+940.497319924"
Oct 29 22:58:03.686228 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:58:03.664855    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:58:03.838620 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.838579    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:03.838913 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.838880    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:03.884174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.840723    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:03.924302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.924242    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:03.924572 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:03.855239    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.104181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.100398    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.180172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.145273    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.180172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.151272    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.180172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.167369    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:04.223205 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.178252    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.243067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.178313    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.243067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.178350    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.243067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.178368    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.243067 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.178388    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:58:04.904998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.904956    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.905334 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.905299    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:04.930223 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.930187    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:04.930534 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:04.930492    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:05.206267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.200224    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Oct 29 22:58:05.206267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.201669    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.211280 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.211227    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.235417 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.235370    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.235801 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.235758    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.236971 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.236934    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.238246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.238217    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.239380 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.239317    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.254637 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.254605    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:58:05.380186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.362488    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:58:05.466982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.432501    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.503442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.503410    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:58:05.561736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.561697    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:05.779607 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.779561    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:05.868298 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.868258    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:05.947078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.947029    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:05.947552 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.868568    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:05.970225 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:05.949791    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:06.020626 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.009352    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:06.229197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.226472    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:06.239167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.236735    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:58:06.239167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.237069    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 29 22:58:06.239167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.238390    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:06.240695 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.240201    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:06.256366 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.250363    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:58:06.265758 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.265737    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:58:06.266095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.266072    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:58:06.267475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.267452    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:58:06.268045 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.268029    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:58:06.272173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.268201    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:58:06.840729 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.840676    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:06.840949 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.840741    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:06.842654 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.842619    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:06.842802 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:06.842668    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:07.403173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.402782    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"]
Oct 29 22:58:07.416875 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.416837    4839 cpu_manager.go:414] "containerMap: removing container" podUID="198ed509-6ae3-4382-b9b1-16065f113c0c" containerName="revoke-keys"
Oct 29 22:58:07.417110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.417091    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198ed509-6ae3-4382-b9b1-16065f113c0c" containerName="revoke-keys"
Oct 29 22:58:07.417418 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.417402    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="198ed509-6ae3-4382-b9b1-16065f113c0c" containerName="revoke-keys"
Oct 29 22:58:07.432114 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.432084    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"]
Oct 29 22:58:07.432344 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.432280    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.452164 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.452101    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:07.452338 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.452181    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:07.515199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.501035    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:07.566329 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.566283    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.567128 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.567099    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.567631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.567607    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsfnm\" (UniqueName: \"kubernetes.io/projected/114aa2b3-f96b-4429-ad1b-27c9e4442095-kube-api-access-gsfnm\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.715543 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.714438    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.741240 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.715958    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.758187 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.715470    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.758187 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.741728    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsfnm\" (UniqueName: \"kubernetes.io/projected/114aa2b3-f96b-4429-ad1b-27c9e4442095-kube-api-access-gsfnm\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.758187 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.736004    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.768367 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.768321    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsfnm\" (UniqueName: \"kubernetes.io/projected/114aa2b3-f96b-4429-ad1b-27c9e4442095-kube-api-access-gsfnm\") pod \"maas-api-revoke-sub-6d91cde7-ll9zq\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:07.830843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.830662    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:07.830843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.830729    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:07.841263 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.841229    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:07.841448 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:07.841296    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:08.054074 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:08.053834    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:08.441265 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:58:08.441229    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114aa2b3_f96b_4429_ad1b_27c9e4442095.slice/crio-54dfa1cda49ab034d424a2293c07d95c0d87931ba60a752cb05264ae3e29bbb0 WatchSource:0}: Error finding container 54dfa1cda49ab034d424a2293c07d95c0d87931ba60a752cb05264ae3e29bbb0: Status 404 returned error can't find the container with id 54dfa1cda49ab034d424a2293c07d95c0d87931ba60a752cb05264ae3e29bbb0
Oct 29 22:58:08.441822 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:08.441781    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"]
Oct 29 22:58:08.830463 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:08.830427    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:08.830653 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:08.830484    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:08.841727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:08.841690    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:08.841905 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:08.841752    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:09.340955 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:09.340920    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq" event={"ID":"114aa2b3-f96b-4429-ad1b-27c9e4442095","Type":"ContainerStarted","Data":"52850aeabeef7586220eeeac6106433cbe8fb5f6daa4be3d4ea18e43fbcb7d7b"}
Oct 29 22:58:09.341244 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:09.341224    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq" event={"ID":"114aa2b3-f96b-4429-ad1b-27c9e4442095","Type":"ContainerStarted","Data":"54dfa1cda49ab034d424a2293c07d95c0d87931ba60a752cb05264ae3e29bbb0"}
Oct 29 22:58:09.422212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:09.422117    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:09.830628 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:09.830596    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:09.831027 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:09.830652    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:09.840541 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:09.840507    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:09.840688 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:09.840559    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:10.350943 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:10.350901    4839 generic.go:356] "Generic (PLEG): container finished" podID="114aa2b3-f96b-4429-ad1b-27c9e4442095" containerID="52850aeabeef7586220eeeac6106433cbe8fb5f6daa4be3d4ea18e43fbcb7d7b" exitCode=0
Oct 29 22:58:10.351170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:10.350980    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq" event={"ID":"114aa2b3-f96b-4429-ad1b-27c9e4442095","Type":"ContainerDied","Data":"52850aeabeef7586220eeeac6106433cbe8fb5f6daa4be3d4ea18e43fbcb7d7b"}
Oct 29 22:58:10.830008 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:10.829963    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 29 22:58:10.830219 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:10.830034    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 29 22:58:10.841363 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:10.841327    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body=
Oct 29 22:58:10.841711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:10.841394    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused"
Oct 29 22:58:11.216723 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.216624    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:11.707181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.707124    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:11.811624 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.811592    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114aa2b3-f96b-4429-ad1b-27c9e4442095-kube-api-access-gsfnm\" (UniqueName: \"kubernetes.io/projected/114aa2b3-f96b-4429-ad1b-27c9e4442095-kube-api-access-gsfnm\") pod \"114aa2b3-f96b-4429-ad1b-27c9e4442095\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") "
Oct 29 22:58:11.811829 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.811694    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-openshift-ca\") pod \"114aa2b3-f96b-4429-ad1b-27c9e4442095\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") "
Oct 29 22:58:11.811829 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.811785    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-xks-ca\") pod \"114aa2b3-f96b-4429-ad1b-27c9e4442095\" (UID: \"114aa2b3-f96b-4429-ad1b-27c9e4442095\") "
Oct 29 22:58:11.812049 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.812025    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-openshift-ca" pod "114aa2b3-f96b-4429-ad1b-27c9e4442095" (UID: "114aa2b3-f96b-4429-ad1b-27c9e4442095"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:11.812352 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.812332    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.813770 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.813745    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114aa2b3-f96b-4429-ad1b-27c9e4442095-kube-api-access-gsfnm" pod "114aa2b3-f96b-4429-ad1b-27c9e4442095" (UID: "114aa2b3-f96b-4429-ad1b-27c9e4442095"). InnerVolumeSpecName "kube-api-access-gsfnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:11.813864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.813821    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-xks-ca" pod "114aa2b3-f96b-4429-ad1b-27c9e4442095" (UID: "114aa2b3-f96b-4429-ad1b-27c9e4442095"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:11.830140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.830115    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:58:11.840326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.840297    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:58:11.845010 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.844986    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:58:11.875195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.875165    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:58:11.913867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.913830    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsfnm\" (UniqueName: \"kubernetes.io/projected/114aa2b3-f96b-4429-ad1b-27c9e4442095-kube-api-access-gsfnm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:11.913867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:11.913864    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/114aa2b3-f96b-4429-ad1b-27c9e4442095-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:12.480557 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:12.480529    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"
Oct 29 22:58:12.480735 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:12.480565    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq" event={"ID":"114aa2b3-f96b-4429-ad1b-27c9e4442095","Type":"ContainerDied","Data":"54dfa1cda49ab034d424a2293c07d95c0d87931ba60a752cb05264ae3e29bbb0"}
Oct 29 22:58:12.480735 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:12.480610    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54dfa1cda49ab034d424a2293c07d95c0d87931ba60a752cb05264ae3e29bbb0"
Oct 29 22:58:12.482023 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:12.481988    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 22:58:12.482353 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:12.482336    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 22:58:13.105167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:13.097521    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:14.407250 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:14.407210    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:14.516812 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:14.516778    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:14.929173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:14.919748    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:15.069775 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:15.069735    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:15.253568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:15.253483    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:16.452678 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:16.452604    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:16.582313 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:16.582274    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:16.976436 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:16.976389    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:17.375717 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.375671    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"]
Oct 29 22:58:17.400065 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.396723    4839 cpu_manager.go:414] "containerMap: removing container" podUID="114aa2b3-f96b-4429-ad1b-27c9e4442095" containerName="revoke-keys"
Oct 29 22:58:17.400065 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.396753    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114aa2b3-f96b-4429-ad1b-27c9e4442095" containerName="revoke-keys"
Oct 29 22:58:17.400065 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.397115    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="114aa2b3-f96b-4429-ad1b-27c9e4442095" containerName="revoke-keys"
Oct 29 22:58:17.406501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.406474    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"]
Oct 29 22:58:17.406765 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.406748    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.452176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.447887    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:17.471419 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.471366    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:17.475489 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.475444    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.475764 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.475731    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4xt\" (UniqueName: \"kubernetes.io/projected/738e6ea5-7781-4fb9-a474-bba91c3906ab-kube-api-access-rk4xt\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.475924 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.475903    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.555644 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.555606    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:17.577686 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.577653    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.577872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.577705    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4xt\" (UniqueName: \"kubernetes.io/projected/738e6ea5-7781-4fb9-a474-bba91c3906ab-kube-api-access-rk4xt\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.577872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.577738    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.578455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.578422    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.580048 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.580026    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.602588 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.602553    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4xt\" (UniqueName: \"kubernetes.io/projected/738e6ea5-7781-4fb9-a474-bba91c3906ab-kube-api-access-rk4xt\") pod \"maas-api-revoke-sub-4b5c2efd-b7plc\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:17.724497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:17.724407    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:18.113394 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:18.113354    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"]
Oct 29 22:58:18.440331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:18.440298    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:18.613171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:18.604443    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc" event={"ID":"738e6ea5-7781-4fb9-a474-bba91c3906ab","Type":"ContainerStarted","Data":"96375467b597b995362f7f0963ccf01d269c71b84ae5708f7cbc98a091efe766"}
Oct 29 22:58:18.613171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:18.604515    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc" event={"ID":"738e6ea5-7781-4fb9-a474-bba91c3906ab","Type":"ContainerStarted","Data":"a3e75a5b0efe855f36b0b68ab769323f8d9db1fc81c6828486a8f40de7103133"}
Oct 29 22:58:18.649940 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:18.649837    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc" podStartSLOduration=1.649822463 podStartE2EDuration="1.649822463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:18.648755479 +0000 UTC m=+955.751831751" watchObservedRunningTime="2026-10-29 22:58:18.649822463 +0000 UTC m=+955.752898734"
Oct 29 22:58:19.422990 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.422892    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"]
Oct 29 22:58:19.427639 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.427610    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.447513 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.447474    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"]
Oct 29 22:58:19.520687 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.520653    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh5n5\" (UniqueName: \"kubernetes.io/projected/1db9a696-83e2-4db4-9d29-b02eefb561c8-kube-api-access-rh5n5\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.520864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.520797    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.520864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.520849    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.539585 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.539550    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:19.613265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.613227    4839 generic.go:356] "Generic (PLEG): container finished" podID="738e6ea5-7781-4fb9-a474-bba91c3906ab" containerID="96375467b597b995362f7f0963ccf01d269c71b84ae5708f7cbc98a091efe766" exitCode=0
Oct 29 22:58:19.613682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.613317    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc" event={"ID":"738e6ea5-7781-4fb9-a474-bba91c3906ab","Type":"ContainerDied","Data":"96375467b597b995362f7f0963ccf01d269c71b84ae5708f7cbc98a091efe766"}
Oct 29 22:58:19.621982 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.621952    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh5n5\" (UniqueName: \"kubernetes.io/projected/1db9a696-83e2-4db4-9d29-b02eefb561c8-kube-api-access-rh5n5\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.622232 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.622101    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.622337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.622260    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.628872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.623213    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.628872 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.625637    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.654130 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.654098    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh5n5\" (UniqueName: \"kubernetes.io/projected/1db9a696-83e2-4db4-9d29-b02eefb561c8-kube-api-access-rh5n5\") pod \"maas-api-revoke-sub-806a7c55-4mkqz\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.742222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.742120    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:19.872494 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:19.872457    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:20.051317 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:58:20.051285    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db9a696_83e2_4db4_9d29_b02eefb561c8.slice/crio-553ca4433e397d3cd2302691d68b9365acbd924baa6ee1dcf94e580a109c8bd6 WatchSource:0}: Error finding container 553ca4433e397d3cd2302691d68b9365acbd924baa6ee1dcf94e580a109c8bd6: Status 404 returned error can't find the container with id 553ca4433e397d3cd2302691d68b9365acbd924baa6ee1dcf94e580a109c8bd6
Oct 29 22:58:20.055712 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:20.055685    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"]
Oct 29 22:58:20.622924 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:20.622887    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz" event={"ID":"1db9a696-83e2-4db4-9d29-b02eefb561c8","Type":"ContainerStarted","Data":"bf020bb67a26315d7f4879aa965b2d233c7b0b342eb9dcd6574373df7be32a0f"}
Oct 29 22:58:20.623396 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:20.622941    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz" event={"ID":"1db9a696-83e2-4db4-9d29-b02eefb561c8","Type":"ContainerStarted","Data":"553ca4433e397d3cd2302691d68b9365acbd924baa6ee1dcf94e580a109c8bd6"}
Oct 29 22:58:20.625625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:20.625587    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:20.901609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:20.901581    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:20.948797 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:20.948742    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz" podStartSLOduration=1.948727536 podStartE2EDuration="1.948727536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:20.731009437 +0000 UTC m=+957.834085709" watchObservedRunningTime="2026-10-29 22:58:20.948727536 +0000 UTC m=+958.051803847"
Oct 29 22:58:21.043697 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.043663    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738e6ea5-7781-4fb9-a474-bba91c3906ab-kube-api-access-rk4xt\" (UniqueName: \"kubernetes.io/projected/738e6ea5-7781-4fb9-a474-bba91c3906ab-kube-api-access-rk4xt\") pod \"738e6ea5-7781-4fb9-a474-bba91c3906ab\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") "
Oct 29 22:58:21.043880 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.043791    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-openshift-ca\") pod \"738e6ea5-7781-4fb9-a474-bba91c3906ab\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") "
Oct 29 22:58:21.043935 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.043915    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-xks-ca\") pod \"738e6ea5-7781-4fb9-a474-bba91c3906ab\" (UID: \"738e6ea5-7781-4fb9-a474-bba91c3906ab\") "
Oct 29 22:58:21.044250 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.044215    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-openshift-ca" pod "738e6ea5-7781-4fb9-a474-bba91c3906ab" (UID: "738e6ea5-7781-4fb9-a474-bba91c3906ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:21.044556 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.044538    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.068755 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.068581    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738e6ea5-7781-4fb9-a474-bba91c3906ab-kube-api-access-rk4xt" pod "738e6ea5-7781-4fb9-a474-bba91c3906ab" (UID: "738e6ea5-7781-4fb9-a474-bba91c3906ab"). InnerVolumeSpecName "kube-api-access-rk4xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:21.068755 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.068635    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-xks-ca" pod "738e6ea5-7781-4fb9-a474-bba91c3906ab" (UID: "738e6ea5-7781-4fb9-a474-bba91c3906ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:21.145651 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.145615    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4xt\" (UniqueName: \"kubernetes.io/projected/738e6ea5-7781-4fb9-a474-bba91c3906ab-kube-api-access-rk4xt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.145651 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.145644    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/738e6ea5-7781-4fb9-a474-bba91c3906ab-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:21.240376 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.240327    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:21.442558 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.442479    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:21.903823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.903785    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc" event={"ID":"738e6ea5-7781-4fb9-a474-bba91c3906ab","Type":"ContainerDied","Data":"a3e75a5b0efe855f36b0b68ab769323f8d9db1fc81c6828486a8f40de7103133"}
Oct 29 22:58:21.904554 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.904534    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e75a5b0efe855f36b0b68ab769323f8d9db1fc81c6828486a8f40de7103133"
Oct 29 22:58:21.904795 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.904749    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"
Oct 29 22:58:21.911027 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.910995    4839 generic.go:356] "Generic (PLEG): container finished" podID="1db9a696-83e2-4db4-9d29-b02eefb561c8" containerID="bf020bb67a26315d7f4879aa965b2d233c7b0b342eb9dcd6574373df7be32a0f" exitCode=0
Oct 29 22:58:21.911172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:21.911036    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz" event={"ID":"1db9a696-83e2-4db4-9d29-b02eefb561c8","Type":"ContainerDied","Data":"bf020bb67a26315d7f4879aa965b2d233c7b0b342eb9dcd6574373df7be32a0f"}
Oct 29 22:58:22.526984 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:22.526953    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:22.690207 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:22.685716    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:22.965951 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:22.965903    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:23.318173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.308257    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:23.443316 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.443278    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:23.607653 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.607573    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:23.720705 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.720650    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-openshift-ca\") pod \"1db9a696-83e2-4db4-9d29-b02eefb561c8\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") "
Oct 29 22:58:23.720910 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.720853    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db9a696-83e2-4db4-9d29-b02eefb561c8-kube-api-access-rh5n5\" (UniqueName: \"kubernetes.io/projected/1db9a696-83e2-4db4-9d29-b02eefb561c8-kube-api-access-rh5n5\") pod \"1db9a696-83e2-4db4-9d29-b02eefb561c8\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") "
Oct 29 22:58:23.727524 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.721082    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-xks-ca\") pod \"1db9a696-83e2-4db4-9d29-b02eefb561c8\" (UID: \"1db9a696-83e2-4db4-9d29-b02eefb561c8\") "
Oct 29 22:58:23.727524 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.721598    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-openshift-ca" pod "1db9a696-83e2-4db4-9d29-b02eefb561c8" (UID: "1db9a696-83e2-4db4-9d29-b02eefb561c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:23.727524 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.725830    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:23.806167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.802898    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-xks-ca" pod "1db9a696-83e2-4db4-9d29-b02eefb561c8" (UID: "1db9a696-83e2-4db4-9d29-b02eefb561c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:23.816099 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.814328    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db9a696-83e2-4db4-9d29-b02eefb561c8-kube-api-access-rh5n5" pod "1db9a696-83e2-4db4-9d29-b02eefb561c8" (UID: "1db9a696-83e2-4db4-9d29-b02eefb561c8"). InnerVolumeSpecName "kube-api-access-rh5n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:23.834169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.828076    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9a696-83e2-4db4-9d29-b02eefb561c8-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:23.834169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.828117    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh5n5\" (UniqueName: \"kubernetes.io/projected/1db9a696-83e2-4db4-9d29-b02eefb561c8-kube-api-access-rh5n5\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:23.980246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:23.980135    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:24.190173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:24.177613    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:24.190348 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:24.190247    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.426168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:24.420725    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz" event={"ID":"1db9a696-83e2-4db4-9d29-b02eefb561c8","Type":"ContainerDied","Data":"553ca4433e397d3cd2302691d68b9365acbd924baa6ee1dcf94e580a109c8bd6"}
Oct 29 22:58:24.426168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:24.420787    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="553ca4433e397d3cd2302691d68b9365acbd924baa6ee1dcf94e580a109c8bd6"
Oct 29 22:58:24.426168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:24.420867    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"
Oct 29 22:58:24.695233 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:24.694714    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:24.986356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:24.986243    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:25.175172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.172472    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:25.227175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.224264    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:25.227175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.224672    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:25.227175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.225530    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:58:25.227175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.225588    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:58:25.245631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.245534    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:58:25.246019 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.245878    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:25.246499 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.245906    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:58:25.261367 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.261319    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:58:25.809177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.802858    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 29 22:58:25.809811 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.809757    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:58:25.836837 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.836802    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:58:26.000200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.995617    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.000200 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:25.998390    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:58:26.010184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.003222    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.117229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.113033    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.234171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.228692    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.234171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.232322    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.246948 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.246912    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.247858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.247838    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:58:26.249990 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.249967    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.251242 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.251224    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:58:26.261930 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.261893    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.263574 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.263549    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:58:26.529562 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.529521    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:26.865231 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.858869    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:26.971823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.970585    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:58:26.996385 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:26.996340    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:58:27.097046 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.097018    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:58:27.138349 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.138244    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:27.152953 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.139287    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:58:27.245875 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.245833    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:27.250434 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.250407    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:58:27.251722 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.251694    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:27.275773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.275746    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:58:27.280653 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.280630    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:58:27.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.353228    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"]
Oct 29 22:58:27.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.354136    4839 cpu_manager.go:414] "containerMap: removing container" podUID="738e6ea5-7781-4fb9-a474-bba91c3906ab" containerName="revoke-keys"
Oct 29 22:58:27.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.354179    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738e6ea5-7781-4fb9-a474-bba91c3906ab" containerName="revoke-keys"
Oct 29 22:58:27.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.354212    4839 cpu_manager.go:414] "containerMap: removing container" podUID="1db9a696-83e2-4db4-9d29-b02eefb561c8" containerName="revoke-keys"
Oct 29 22:58:27.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.354221    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db9a696-83e2-4db4-9d29-b02eefb561c8" containerName="revoke-keys"
Oct 29 22:58:27.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.354412    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="738e6ea5-7781-4fb9-a474-bba91c3906ab" containerName="revoke-keys"
Oct 29 22:58:27.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.354428    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db9a696-83e2-4db4-9d29-b02eefb561c8" containerName="revoke-keys"
Oct 29 22:58:27.366777 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.365460    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.381054 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.381022    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"]
Oct 29 22:58:27.391852 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.384331    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:58:27.405577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.403515    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:27.405577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.403584    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:27.434633 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.434602    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:58:27.435285 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.435265    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:58:27.435699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.435670    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.435885 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.435866    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.436048 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.436032    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rjh\" (UniqueName: \"kubernetes.io/projected/c41ba44f-c6c6-4e64-95d7-dafec8499831-kube-api-access-64rjh\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.537233 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.537202    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.537443 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.537290    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.537443 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.537325    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rjh\" (UniqueName: \"kubernetes.io/projected/c41ba44f-c6c6-4e64-95d7-dafec8499831-kube-api-access-64rjh\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.538640 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.538609    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.550487 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.540559    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.560034 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.560001    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rjh\" (UniqueName: \"kubernetes.io/projected/c41ba44f-c6c6-4e64-95d7-dafec8499831-kube-api-access-64rjh\") pod \"maas-api-revoke-sub-7d0ab918-fxh6s\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:27.736340 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:27.736238    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:28.041441 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:58:28.041412    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41ba44f_c6c6_4e64_95d7_dafec8499831.slice/crio-3ba90205b6232a82add6e7453bc012abf957c259493ec8a50190c0211ae7fd47 WatchSource:0}: Error finding container 3ba90205b6232a82add6e7453bc012abf957c259493ec8a50190c0211ae7fd47: Status 404 returned error can't find the container with id 3ba90205b6232a82add6e7453bc012abf957c259493ec8a50190c0211ae7fd47
Oct 29 22:58:28.043530 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:28.043502    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"]
Oct 29 22:58:28.320733 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:28.320658    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:28.422120 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:28.422091    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:28.611599 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:28.611521    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:28.643305 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:28.643269    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s" event={"ID":"c41ba44f-c6c6-4e64-95d7-dafec8499831","Type":"ContainerStarted","Data":"d0fb0b52883cec945ba4c2650e1cbff591635c6796adc763554bed51fae2b08d"}
Oct 29 22:58:28.643475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:28.643326    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s" event={"ID":"c41ba44f-c6c6-4e64-95d7-dafec8499831","Type":"ContainerStarted","Data":"3ba90205b6232a82add6e7453bc012abf957c259493ec8a50190c0211ae7fd47"}
Oct 29 22:58:28.700498 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:28.700305    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s" podStartSLOduration=1.700285445 podStartE2EDuration="1.700285445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:28.694861609 +0000 UTC m=+965.797937880" watchObservedRunningTime="2026-10-29 22:58:28.700285445 +0000 UTC m=+965.803361717"
Oct 29 22:58:29.653990 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:29.653957    4839 generic.go:356] "Generic (PLEG): container finished" podID="c41ba44f-c6c6-4e64-95d7-dafec8499831" containerID="d0fb0b52883cec945ba4c2650e1cbff591635c6796adc763554bed51fae2b08d" exitCode=0
Oct 29 22:58:29.654448 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:29.654183    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s" event={"ID":"c41ba44f-c6c6-4e64-95d7-dafec8499831","Type":"ContainerDied","Data":"d0fb0b52883cec945ba4c2650e1cbff591635c6796adc763554bed51fae2b08d"}
Oct 29 22:58:29.842473 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:29.842441    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:30.115624 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.115591    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:30.139207 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.139171    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"]
Oct 29 22:58:30.168297 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.144749    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.210172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.177355    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-01e33b3d\""
Oct 29 22:58:30.210172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.178863    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27404b4f-4283-4e65-bd72-c366b6751331-plugins-config-volume\") pod \"payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") " pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.210172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.193018    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-01e33b3d-dockercfg-266tp\""
Oct 29 22:58:30.210553 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.210520    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts99k\" (UniqueName: \"kubernetes.io/projected/27404b4f-4283-4e65-bd72-c366b6751331-kube-api-access-ts99k\") pod \"payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") " pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.331229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.316546    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27404b4f-4283-4e65-bd72-c366b6751331-plugins-config-volume\") pod \"payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") " pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.366221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.366116    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts99k\" (UniqueName: \"kubernetes.io/projected/27404b4f-4283-4e65-bd72-c366b6751331-kube-api-access-ts99k\") pod \"payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") " pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.413170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.358174    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27404b4f-4283-4e65-bd72-c366b6751331-plugins-config-volume\") pod \"payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") " pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.473268 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.473225    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"]
Oct 29 22:58:30.505234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.505130    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:30.508747 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.508668    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts99k\" (UniqueName: \"kubernetes.io/projected/27404b4f-4283-4e65-bd72-c366b6751331-kube-api-access-ts99k\") pod \"payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") " pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.571332 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.571301    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:30.694215 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.689696    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:30.920820 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:30.920790    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:31.074186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.069576    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:31.092216 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.092174    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"]
Oct 29 22:58:31.139195 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.139162    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-xks-ca\") pod \"c41ba44f-c6c6-4e64-95d7-dafec8499831\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") "
Oct 29 22:58:31.139359 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.139309    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41ba44f-c6c6-4e64-95d7-dafec8499831-kube-api-access-64rjh\" (UniqueName: \"kubernetes.io/projected/c41ba44f-c6c6-4e64-95d7-dafec8499831-kube-api-access-64rjh\") pod \"c41ba44f-c6c6-4e64-95d7-dafec8499831\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") "
Oct 29 22:58:31.139510 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.139492    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-openshift-ca\") pod \"c41ba44f-c6c6-4e64-95d7-dafec8499831\" (UID: \"c41ba44f-c6c6-4e64-95d7-dafec8499831\") "
Oct 29 22:58:31.140090 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.139910    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-openshift-ca" pod "c41ba44f-c6c6-4e64-95d7-dafec8499831" (UID: "c41ba44f-c6c6-4e64-95d7-dafec8499831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:31.140090 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.140065    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:31.142199 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.142176    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-xks-ca" pod "c41ba44f-c6c6-4e64-95d7-dafec8499831" (UID: "c41ba44f-c6c6-4e64-95d7-dafec8499831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:31.143544 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.143522    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41ba44f-c6c6-4e64-95d7-dafec8499831-kube-api-access-64rjh" pod "c41ba44f-c6c6-4e64-95d7-dafec8499831" (UID: "c41ba44f-c6c6-4e64-95d7-dafec8499831"). InnerVolumeSpecName "kube-api-access-64rjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:31.145032 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.145007    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:31.241416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.241343    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41ba44f-c6c6-4e64-95d7-dafec8499831-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:31.241416 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.241370    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rjh\" (UniqueName: \"kubernetes.io/projected/c41ba44f-c6c6-4e64-95d7-dafec8499831-kube-api-access-64rjh\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:31.753408 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.753365    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s" event={"ID":"c41ba44f-c6c6-4e64-95d7-dafec8499831","Type":"ContainerDied","Data":"3ba90205b6232a82add6e7453bc012abf957c259493ec8a50190c0211ae7fd47"}
Oct 29 22:58:31.753895 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.753428    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ba90205b6232a82add6e7453bc012abf957c259493ec8a50190c0211ae7fd47"
Oct 29 22:58:31.753895 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.753376    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"
Oct 29 22:58:31.755324 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.755286    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw" event={"ID":"27404b4f-4283-4e65-bd72-c366b6751331","Type":"ContainerStarted","Data":"da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c"}
Oct 29 22:58:31.755481 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.755338    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw" event={"ID":"27404b4f-4283-4e65-bd72-c366b6751331","Type":"ContainerStarted","Data":"e208c6f524f0b2c2ecf798ff9d7bdd2ae5ea618910333fb7475a9775a12134ca"}
Oct 29 22:58:31.755564 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.755550    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:31.801691 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:31.801601    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw" podStartSLOduration=1.801588899 podStartE2EDuration="1.801588899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:31.798955588 +0000 UTC m=+968.902031872" watchObservedRunningTime="2026-10-29 22:58:31.801588899 +0000 UTC m=+968.904665169"
Oct 29 22:58:32.052760 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.052673    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"]
Oct 29 22:58:32.053418 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.053396    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c41ba44f-c6c6-4e64-95d7-dafec8499831" containerName="revoke-keys"
Oct 29 22:58:32.053418 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.053418    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41ba44f-c6c6-4e64-95d7-dafec8499831" containerName="revoke-keys"
Oct 29 22:58:32.053594 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.053545    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41ba44f-c6c6-4e64-95d7-dafec8499831" containerName="revoke-keys"
Oct 29 22:58:32.057424 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.057400    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.082186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.081439    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"]
Oct 29 22:58:32.117961 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.117932    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:32.118135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.118048    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:32.178013 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.177973    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cgh\" (UniqueName: \"kubernetes.io/projected/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-kube-api-access-l7cgh\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.178226 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.178029    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.178393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.178361    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.279576 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.279545    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.279750 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.279663    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cgh\" (UniqueName: \"kubernetes.io/projected/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-kube-api-access-l7cgh\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.279750 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.279696    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.280410 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.280385    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.282017 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.281989    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.300534 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.300506    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cgh\" (UniqueName: \"kubernetes.io/projected/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-kube-api-access-l7cgh\") pod \"maas-api-revoke-sub-f5eb311b-brrkc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.383210 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.383102    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:32.584847 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.584814    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:32.749258 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:32.737748    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"]
Oct 29 22:58:32.761251 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:58:32.761208    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ce1bbb_d479_4934_b7c9_8d3e128a06cc.slice/crio-81a60aa0cb4fff103c8817e840247f2e03b0b6a1ff665c730f8c369d39b86dac WatchSource:0}: Error finding container 81a60aa0cb4fff103c8817e840247f2e03b0b6a1ff665c730f8c369d39b86dac: Status 404 returned error can't find the container with id 81a60aa0cb4fff103c8817e840247f2e03b0b6a1ff665c730f8c369d39b86dac
Oct 29 22:58:33.238477 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:58:33.238434    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:58:33.779783 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:33.779743    4839 generic.go:356] "Generic (PLEG): container finished" podID="42ce1bbb-d479-4934-b7c9-8d3e128a06cc" containerID="4ae1e6a27d00a63e3eb83adf0251e32553dd4c5e2a8826ceea0254d00a30473f" exitCode=0
Oct 29 22:58:33.780220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:33.779826    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc" event={"ID":"42ce1bbb-d479-4934-b7c9-8d3e128a06cc","Type":"ContainerDied","Data":"4ae1e6a27d00a63e3eb83adf0251e32553dd4c5e2a8826ceea0254d00a30473f"}
Oct 29 22:58:33.780220 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:33.779900    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc" event={"ID":"42ce1bbb-d479-4934-b7c9-8d3e128a06cc","Type":"ContainerStarted","Data":"81a60aa0cb4fff103c8817e840247f2e03b0b6a1ff665c730f8c369d39b86dac"}
Oct 29 22:58:34.298922 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:34.298885    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:34.946682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:34.946658    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:35.022319 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.022290    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-kube-api-access-l7cgh\" (UniqueName: \"kubernetes.io/projected/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-kube-api-access-l7cgh\") pod \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") "
Oct 29 22:58:35.022505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.022342    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-xks-ca\") pod \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") "
Oct 29 22:58:35.022505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.022429    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-openshift-ca\") pod \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\" (UID: \"42ce1bbb-d479-4934-b7c9-8d3e128a06cc\") "
Oct 29 22:58:35.022869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.022833    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-openshift-ca" pod "42ce1bbb-d479-4934-b7c9-8d3e128a06cc" (UID: "42ce1bbb-d479-4934-b7c9-8d3e128a06cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:35.023337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.023309    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:35.024757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.024724    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-kube-api-access-l7cgh" pod "42ce1bbb-d479-4934-b7c9-8d3e128a06cc" (UID: "42ce1bbb-d479-4934-b7c9-8d3e128a06cc"). InnerVolumeSpecName "kube-api-access-l7cgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:35.024908 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.024887    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-xks-ca" pod "42ce1bbb-d479-4934-b7c9-8d3e128a06cc" (UID: "42ce1bbb-d479-4934-b7c9-8d3e128a06cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:35.124773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.124692    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7cgh\" (UniqueName: \"kubernetes.io/projected/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-kube-api-access-l7cgh\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:35.124773 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.124723    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42ce1bbb-d479-4934-b7c9-8d3e128a06cc-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:35.805309 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.805267    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"
Oct 29 22:58:35.805483 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.805311    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc" event={"ID":"42ce1bbb-d479-4934-b7c9-8d3e128a06cc","Type":"ContainerDied","Data":"81a60aa0cb4fff103c8817e840247f2e03b0b6a1ff665c730f8c369d39b86dac"}
Oct 29 22:58:35.805483 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:35.805359    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81a60aa0cb4fff103c8817e840247f2e03b0b6a1ff665c730f8c369d39b86dac"
Oct 29 22:58:36.092547 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.092430    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"]
Oct 29 22:58:36.093189 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.093171    4839 cpu_manager.go:414] "containerMap: removing container" podUID="42ce1bbb-d479-4934-b7c9-8d3e128a06cc" containerName="revoke-keys"
Oct 29 22:58:36.093247 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.093190    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ce1bbb-d479-4934-b7c9-8d3e128a06cc" containerName="revoke-keys"
Oct 29 22:58:36.093341 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.093331    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ce1bbb-d479-4934-b7c9-8d3e128a06cc" containerName="revoke-keys"
Oct 29 22:58:36.097441 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.097411    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.129336 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.129288    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"]
Oct 29 22:58:36.130816 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.130457    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2bd3f07e-serving-cert\""
Oct 29 22:58:36.139515 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.139476    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-maas-api-tls\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.139711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.139540    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-metrics-certs\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.139711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.139560    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvld2\" (UniqueName: \"kubernetes.io/projected/1cb0f1d1-f491-49d1-a74b-99db2d490997-kube-api-access-tvld2\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.139711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.139593    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0f1d1-f491-49d1-a74b-99db2d490997-openshift-service-ca\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.199482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.199441    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:36.241094 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.241054    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-maas-api-tls\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.241355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.241122    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-metrics-certs\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.241355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.241193    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvld2\" (UniqueName: \"kubernetes.io/projected/1cb0f1d1-f491-49d1-a74b-99db2d490997-kube-api-access-tvld2\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.241355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.241241    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0f1d1-f491-49d1-a74b-99db2d490997-openshift-service-ca\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.241925 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.241900    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0f1d1-f491-49d1-a74b-99db2d490997-openshift-service-ca\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.243864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.243838    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-metrics-certs\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.243981 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.243947    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-maas-api-tls\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.294418 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.294343    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvld2\" (UniqueName: \"kubernetes.io/projected/1cb0f1d1-f491-49d1-a74b-99db2d490997-kube-api-access-tvld2\") pod \"maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.356520 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.356412    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:36.416992 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.416956    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:36.773161 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:58:36.773115    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb0f1d1_f491_49d1_a74b_99db2d490997.slice/crio-3ede2ea3e834c984842de7b056127524743a29495a8dd33eb1098ebcfab49f43 WatchSource:0}: Error finding container 3ede2ea3e834c984842de7b056127524743a29495a8dd33eb1098ebcfab49f43: Status 404 returned error can't find the container with id 3ede2ea3e834c984842de7b056127524743a29495a8dd33eb1098ebcfab49f43
Oct 29 22:58:36.774681 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.774659    4839 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 22:58:36.775071 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.775048    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"]
Oct 29 22:58:36.814202 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:36.814138    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg" event={"ID":"1cb0f1d1-f491-49d1-a74b-99db2d490997","Type":"ContainerStarted","Data":"3ede2ea3e834c984842de7b056127524743a29495a8dd33eb1098ebcfab49f43"}
Oct 29 22:58:37.824719 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:37.824669    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg" event={"ID":"1cb0f1d1-f491-49d1-a74b-99db2d490997","Type":"ContainerStarted","Data":"4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b"}
Oct 29 22:58:37.825093 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:37.824769    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:37.874040 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:37.873993    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg" podStartSLOduration=1.510554787 podStartE2EDuration="1.873978866s" totalImagesPullingTime="363.424079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:37.868732389 +0000 UTC m=+974.971808676" watchObservedRunningTime="2026-10-29 22:58:37.873978866 +0000 UTC m=+974.977055136"
Oct 29 22:58:37.969748 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:37.969709    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:39.507865 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:39.507830    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:41.997497 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:41.990310    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.108172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:42.104975    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:42.125174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:42.119970    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:58:42.153169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:42.146102    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.787174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:42.786437    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:58:42.864544 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:42.863258    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:42.864544 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:42.863757    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.091484 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.091347    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.151291 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.151236    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.187176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.165071    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.187176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.171210    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded"
Oct 29 22:58:43.306339 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.306289    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.312177 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.308882    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.391007 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.313250    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.412827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.412776    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.435081 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.435051    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:58:43.448566 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.448535    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:58:43.449888 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.449863    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:58:43.610197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.601354    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 22:58:43.610197 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.601441    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.833018 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.812940    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.870268 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.866369    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:58:43.870268 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.867434    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:43.886555 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:43.886522    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:58:44.100981 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.100874    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:44.180920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.179522    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.180920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.180046    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.180920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.180408    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.194007 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.192417    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.194007 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.192761    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:58:44.220475 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.218832    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:58:44.316896 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.310542    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded"
Oct 29 22:58:44.316896 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.312989    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:58:44.400117 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.399987    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:44.406095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.406057    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.451490 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.411265    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:58:44.451490 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.413402    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:58:44.470122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.452374    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 22:58:44.470122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.463319    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:58:44.470122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.463951    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:58:44.470122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.464099    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:58:44.470122 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.464261    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:58:44.522971 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.522932    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:58:44.598750 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.598718    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:58:44.602285 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.602257    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 22:58:44.666438 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.666350    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:58:44.693380 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.693346    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:58:44.759186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.757074    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:58:44.805803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.805718    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:44.996657 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:44.996566    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:47.193203 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:47.193163    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:47.356045 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:47.356007    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:48.822641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:48.822607    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:49.863135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:49.863090    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:49.927340 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:49.927300    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"]
Oct 29 22:58:49.932351 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:49.932323    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:49.951096 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:49.951066    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"]
Oct 29 22:58:49.959063 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:49.959036    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:58:49.959063 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:49.959036    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:58:50.053064 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.053022    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.053288 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.053137    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f66lj\" (UniqueName: \"kubernetes.io/projected/d8e5e531-c774-44bc-9dd5-d508f571bd7d-kube-api-access-f66lj\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.053447 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.053422    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.154978 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.154875    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.155174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.155016    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.155174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.155091    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f66lj\" (UniqueName: \"kubernetes.io/projected/d8e5e531-c774-44bc-9dd5-d508f571bd7d-kube-api-access-f66lj\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.155667 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.155647    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.157448 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.157422    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.176377 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.176339    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f66lj\" (UniqueName: \"kubernetes.io/projected/d8e5e531-c774-44bc-9dd5-d508f571bd7d-kube-api-access-f66lj\") pod \"maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.249920 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.249879    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:50.554007 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:58:50.553972    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e5e531_c774_44bc_9dd5_d508f571bd7d.slice/crio-d8b7521b2ab764d0528c2de546c96fc542f6c7faa733a8295b88acf7f863c42e WatchSource:0}: Error finding container d8b7521b2ab764d0528c2de546c96fc542f6c7faa733a8295b88acf7f863c42e: Status 404 returned error can't find the container with id d8b7521b2ab764d0528c2de546c96fc542f6c7faa733a8295b88acf7f863c42e
Oct 29 22:58:50.555753 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.555729    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"]
Oct 29 22:58:50.620104 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:50.620071    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:51.164080 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:51.163972    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m" event={"ID":"d8e5e531-c774-44bc-9dd5-d508f571bd7d","Type":"ContainerStarted","Data":"6f82df9ac2af3f2a7217a6d5fe6d5b242f55cfcc3e25738d36676af00af69f98"}
Oct 29 22:58:51.164080 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:51.164054    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m" event={"ID":"d8e5e531-c774-44bc-9dd5-d508f571bd7d","Type":"ContainerStarted","Data":"d8b7521b2ab764d0528c2de546c96fc542f6c7faa733a8295b88acf7f863c42e"}
Oct 29 22:58:51.209484 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:51.209425    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m" podStartSLOduration=2.209409466 podStartE2EDuration="2.209409466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:58:51.206658949 +0000 UTC m=+988.309735219" watchObservedRunningTime="2026-10-29 22:58:51.209409466 +0000 UTC m=+988.312485782"
Oct 29 22:58:52.174541 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:52.174497    4839 generic.go:356] "Generic (PLEG): container finished" podID="d8e5e531-c774-44bc-9dd5-d508f571bd7d" containerID="6f82df9ac2af3f2a7217a6d5fe6d5b242f55cfcc3e25738d36676af00af69f98" exitCode=0
Oct 29 22:58:52.174989 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:52.174582    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m" event={"ID":"d8e5e531-c774-44bc-9dd5-d508f571bd7d","Type":"ContainerDied","Data":"6f82df9ac2af3f2a7217a6d5fe6d5b242f55cfcc3e25738d36676af00af69f98"}
Oct 29 22:58:52.497465 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:52.497381    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:53.350596 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.350569    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:53.403590 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.403561    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e5e531-c774-44bc-9dd5-d508f571bd7d-kube-api-access-f66lj\" (UniqueName: \"kubernetes.io/projected/d8e5e531-c774-44bc-9dd5-d508f571bd7d-kube-api-access-f66lj\") pod \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") "
Oct 29 22:58:53.403762 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.403674    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-openshift-ca\") pod \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") "
Oct 29 22:58:53.403903 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.403888    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-xks-ca\") pod \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\" (UID: \"d8e5e531-c774-44bc-9dd5-d508f571bd7d\") "
Oct 29 22:58:53.404110 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.404080    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-openshift-ca" pod "d8e5e531-c774-44bc-9dd5-d508f571bd7d" (UID: "d8e5e531-c774-44bc-9dd5-d508f571bd7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:58:53.404530 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.404512    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:53.406485 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.406456    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-xks-ca" pod "d8e5e531-c774-44bc-9dd5-d508f571bd7d" (UID: "d8e5e531-c774-44bc-9dd5-d508f571bd7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:58:53.406589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.406494    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e5e531-c774-44bc-9dd5-d508f571bd7d-kube-api-access-f66lj" pod "d8e5e531-c774-44bc-9dd5-d508f571bd7d" (UID: "d8e5e531-c774-44bc-9dd5-d508f571bd7d"). InnerVolumeSpecName "kube-api-access-f66lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:58:53.506326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.506252    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f66lj\" (UniqueName: \"kubernetes.io/projected/d8e5e531-c774-44bc-9dd5-d508f571bd7d-kube-api-access-f66lj\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:53.506326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:53.506282    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8e5e531-c774-44bc-9dd5-d508f571bd7d-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:58:54.193841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:54.193807    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"
Oct 29 22:58:54.194043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:54.193837    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m" event={"ID":"d8e5e531-c774-44bc-9dd5-d508f571bd7d","Type":"ContainerDied","Data":"d8b7521b2ab764d0528c2de546c96fc542f6c7faa733a8295b88acf7f863c42e"}
Oct 29 22:58:54.194043 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:54.193885    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8b7521b2ab764d0528c2de546c96fc542f6c7faa733a8295b88acf7f863c42e"
Oct 29 22:58:54.206965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:54.206928    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:54.833882 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:54.833844    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"]
Oct 29 22:58:54.845852 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:54.845819    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"]
Oct 29 22:58:55.143594 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:55.143508    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:55.336438 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:55.336402    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:55.989314 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:55.989270    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:56.311770 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:56.311736    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:56.426670 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:56.426632    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:57.426178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:57.414649    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:57.827658 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:57.827627    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:58:59.837169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:58:59.834377    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:00.524432 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.524393    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:00.799125 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.799000    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:00.923171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.920306    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:00.960172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.954752    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"]
Oct 29 22:59:00.960172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.956073    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d8e5e531-c774-44bc-9dd5-d508f571bd7d" containerName="revoke-keys"
Oct 29 22:59:00.960172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.956097    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e5e531-c774-44bc-9dd5-d508f571bd7d" containerName="revoke-keys"
Oct 29 22:59:00.960172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.956286    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e5e531-c774-44bc-9dd5-d508f571bd7d" containerName="revoke-keys"
Oct 29 22:59:00.961701 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.961672    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:00.969208 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.969127    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"]
Oct 29 22:59:00.984654 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.984624    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:00.996438 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:00.996408    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:59:01.070547 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.070455    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.070705 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.070654    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.070802 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.070786    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsdkg\" (UniqueName: \"kubernetes.io/projected/8c0ae155-dfcf-4998-82ab-335d40659825-kube-api-access-hsdkg\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.171942 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.171912    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.172186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.172092    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsdkg\" (UniqueName: \"kubernetes.io/projected/8c0ae155-dfcf-4998-82ab-335d40659825-kube-api-access-hsdkg\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.172304 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.172254    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.172853 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.172829    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.174766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.174746    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.193740 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.193712    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsdkg\" (UniqueName: \"kubernetes.io/projected/8c0ae155-dfcf-4998-82ab-335d40659825-kube-api-access-hsdkg\") pod \"maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.288222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.287468    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:01.741434 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.741396    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:01.844065 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:59:01.844034    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0ae155_dfcf_4998_82ab_335d40659825.slice/crio-9426972d5a9542364442908181e4afc4186d88141a807e1a43b359aef9192947 WatchSource:0}: Error finding container 9426972d5a9542364442908181e4afc4186d88141a807e1a43b359aef9192947: Status 404 returned error can't find the container with id 9426972d5a9542364442908181e4afc4186d88141a807e1a43b359aef9192947
Oct 29 22:59:01.846712 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:01.846680    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"]
Oct 29 22:59:02.328167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:02.322203    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc" event={"ID":"8c0ae155-dfcf-4998-82ab-335d40659825","Type":"ContainerStarted","Data":"9426972d5a9542364442908181e4afc4186d88141a807e1a43b359aef9192947"}
Oct 29 22:59:03.411091 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:03.410994    4839 generic.go:356] "Generic (PLEG): container finished" podID="8c0ae155-dfcf-4998-82ab-335d40659825" containerID="50c05fa09a583a3fb9b4b3e1788eb5aa7a66f8c5821c46500c2443fdd1ddcc82" exitCode=0
Oct 29 22:59:03.411777 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:03.411746    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc" event={"ID":"8c0ae155-dfcf-4998-82ab-335d40659825","Type":"ContainerDied","Data":"50c05fa09a583a3fb9b4b3e1788eb5aa7a66f8c5821c46500c2443fdd1ddcc82"}
Oct 29 22:59:03.464757 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:03.464718    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:59:03.670355 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:03.670244    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:03.712302 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:03.712246    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:03.777291 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:03.777240    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:03.947224 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:03.947123    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.196280 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.196233    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.197288 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.197210    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:59:04.228206 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.221577    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.289171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.281583    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.526398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.525530    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.610377 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.610335    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.626183 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.615162    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:04.671946 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.671890    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.681207 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.672887    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:59:04.716286 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.716238    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:04.725170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.717536    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:59:04.848221 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.823052    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:59:04.980914 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:04.980875    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:59:05.409170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.409087    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.547222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.547173    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.645238 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.644385    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.650476 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.650444    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:59:05.650774 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.650757    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:59:05.650952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.650938    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 22:59:05.651111 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.651098    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:59:05.698158 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.698052    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 29 22:59:05.758972 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.758932    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:05.980227 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.970842    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:05.990799 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:05.990756    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.031172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.011296    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.260706 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.260606    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.298201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.298135    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 29 22:59:06.547358 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.431418    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.553588 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.551372    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.600175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.595804    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.658167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.625603    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.658167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.646236    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:06.658167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.646639    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Oct 29 22:59:06.732275 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:06.732225    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.141582 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.131219    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:59:07.272847 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.272602    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.415931 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.415857    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:07.446652 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.446344    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.454461 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.454431    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 22:59:07.454703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.454689    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:59:07.454827 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.454808    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc" event={"ID":"8c0ae155-dfcf-4998-82ab-335d40659825","Type":"ContainerDied","Data":"9426972d5a9542364442908181e4afc4186d88141a807e1a43b359aef9192947"}
Oct 29 22:59:07.454952 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.454936    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9426972d5a9542364442908181e4afc4186d88141a807e1a43b359aef9192947"
Oct 29 22:59:07.455092 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.455079    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:59:07.455244 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.455231    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:59:07.522569 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.520331    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 22:59:07.544813 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.544778    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-openshift-ca\") pod \"8c0ae155-dfcf-4998-82ab-335d40659825\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") "
Oct 29 22:59:07.545955 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.545919    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-openshift-ca" pod "8c0ae155-dfcf-4998-82ab-335d40659825" (UID: "8c0ae155-dfcf-4998-82ab-335d40659825"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:07.546206 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.546120    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c0ae155-dfcf-4998-82ab-335d40659825-kube-api-access-hsdkg\" (UniqueName: \"kubernetes.io/projected/8c0ae155-dfcf-4998-82ab-335d40659825-kube-api-access-hsdkg\") pod \"8c0ae155-dfcf-4998-82ab-335d40659825\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") "
Oct 29 22:59:07.546883 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.546532    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-xks-ca\") pod \"8c0ae155-dfcf-4998-82ab-335d40659825\" (UID: \"8c0ae155-dfcf-4998-82ab-335d40659825\") "
Oct 29 22:59:07.560687 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.560314    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.592369 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.592334    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:07.611470 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.611440    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0ae155-dfcf-4998-82ab-335d40659825-kube-api-access-hsdkg" pod "8c0ae155-dfcf-4998-82ab-335d40659825" (UID: "8c0ae155-dfcf-4998-82ab-335d40659825"). InnerVolumeSpecName "kube-api-access-hsdkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:07.613893 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.613851    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-xks-ca" pod "8c0ae155-dfcf-4998-82ab-335d40659825" (UID: "8c0ae155-dfcf-4998-82ab-335d40659825"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:07.625957 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.625138    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 22:59:07.654400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.652260    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:07.679682 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.679573    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 22:59:07.698459 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.698423    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsdkg\" (UniqueName: \"kubernetes.io/projected/8c0ae155-dfcf-4998-82ab-335d40659825-kube-api-access-hsdkg\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:07.699331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.699309    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c0ae155-dfcf-4998-82ab-335d40659825-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:07.908123 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:07.908083    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:08.454686 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:08.454656    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"
Oct 29 22:59:09.764756 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:09.764708    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:09.863173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:09.862612    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main" containerID="cri-o://66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b" gracePeriod=45
Oct 29 22:59:09.872725 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:09.868067    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" containerID="cri-o://aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572" gracePeriod=45
Oct 29 22:59:10.192713 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.192688    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:59:10.195995 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.195973    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:59:10.249787 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.249757    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-model-cache\") pod \"6d006a24-6341-4d53-b847-e49793bd0531\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") "
Oct 29 22:59:10.249935 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.249819    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-home\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-home\") pod \"6d006a24-6341-4d53-b847-e49793bd0531\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") "
Oct 29 22:59:10.249935 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.249904    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-model-cache\") pod \"ecc56888-0e26-441a-a3d4-da19bba238ee\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") "
Oct 29 22:59:10.250061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.249994    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-dshm\") pod \"ecc56888-0e26-441a-a3d4-da19bba238ee\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") "
Oct 29 22:59:10.250061 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250025    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-tmp-dir\") pod \"6d006a24-6341-4d53-b847-e49793bd0531\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") "
Oct 29 22:59:10.250186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250062    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-model-cache" pod "6d006a24-6341-4d53-b847-e49793bd0531" (UID: "6d006a24-6341-4d53-b847-e49793bd0531"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.250186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250092    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-tmp-dir\") pod \"ecc56888-0e26-441a-a3d4-da19bba238ee\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") "
Oct 29 22:59:10.250186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250130    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d006a24-6341-4d53-b847-e49793bd0531-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d006a24-6341-4d53-b847-e49793bd0531-tls-certs\") pod \"6d006a24-6341-4d53-b847-e49793bd0531\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") "
Oct 29 22:59:10.250186 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250122    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-home" pod "6d006a24-6341-4d53-b847-e49793bd0531" (UID: "6d006a24-6341-4d53-b847-e49793bd0531"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.250403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250197    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-dshm\") pod \"6d006a24-6341-4d53-b847-e49793bd0531\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") "
Oct 29 22:59:10.250403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250247    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecc56888-0e26-441a-a3d4-da19bba238ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc56888-0e26-441a-a3d4-da19bba238ee-tls-certs\") pod \"ecc56888-0e26-441a-a3d4-da19bba238ee\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") "
Oct 29 22:59:10.250403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250298    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d006a24-6341-4d53-b847-e49793bd0531-kube-api-access-nqp2k\" (UniqueName: \"kubernetes.io/projected/6d006a24-6341-4d53-b847-e49793bd0531-kube-api-access-nqp2k\") pod \"6d006a24-6341-4d53-b847-e49793bd0531\" (UID: \"6d006a24-6341-4d53-b847-e49793bd0531\") "
Oct 29 22:59:10.250403 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250355    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-home\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-home\") pod \"ecc56888-0e26-441a-a3d4-da19bba238ee\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") "
Oct 29 22:59:10.250669 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250399    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-tmp-dir" pod "ecc56888-0e26-441a-a3d4-da19bba238ee" (UID: "ecc56888-0e26-441a-a3d4-da19bba238ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.250669 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250559    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc56888-0e26-441a-a3d4-da19bba238ee-kube-api-access-v8xw4\" (UniqueName: \"kubernetes.io/projected/ecc56888-0e26-441a-a3d4-da19bba238ee-kube-api-access-v8xw4\") pod \"ecc56888-0e26-441a-a3d4-da19bba238ee\" (UID: \"ecc56888-0e26-441a-a3d4-da19bba238ee\") "
Oct 29 22:59:10.250669 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.250590    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-model-cache" pod "ecc56888-0e26-441a-a3d4-da19bba238ee" (UID: "ecc56888-0e26-441a-a3d4-da19bba238ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.251075 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251051    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-home" pod "ecc56888-0e26-441a-a3d4-da19bba238ee" (UID: "ecc56888-0e26-441a-a3d4-da19bba238ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.251215 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251077    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-tmp-dir" pod "6d006a24-6341-4d53-b847-e49793bd0531" (UID: "6d006a24-6341-4d53-b847-e49793bd0531"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.251568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251466    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.251568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251495    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.251568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251510    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.251568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251523    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.251568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251535    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.251568 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.251547    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.252873 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.252848    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-dshm" pod "ecc56888-0e26-441a-a3d4-da19bba238ee" (UID: "ecc56888-0e26-441a-a3d4-da19bba238ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.253021 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.252895    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d006a24-6341-4d53-b847-e49793bd0531-tls-certs" pod "6d006a24-6341-4d53-b847-e49793bd0531" (UID: "6d006a24-6341-4d53-b847-e49793bd0531"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:10.254051 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.254023    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc56888-0e26-441a-a3d4-da19bba238ee-kube-api-access-v8xw4" pod "ecc56888-0e26-441a-a3d4-da19bba238ee" (UID: "ecc56888-0e26-441a-a3d4-da19bba238ee"). InnerVolumeSpecName "kube-api-access-v8xw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:10.254216 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.254198    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d006a24-6341-4d53-b847-e49793bd0531-kube-api-access-nqp2k" pod "6d006a24-6341-4d53-b847-e49793bd0531" (UID: "6d006a24-6341-4d53-b847-e49793bd0531"). InnerVolumeSpecName "kube-api-access-nqp2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:10.254412 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.254390    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-dshm" pod "6d006a24-6341-4d53-b847-e49793bd0531" (UID: "6d006a24-6341-4d53-b847-e49793bd0531"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:10.255496 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.255470    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecc56888-0e26-441a-a3d4-da19bba238ee-tls-certs" pod "ecc56888-0e26-441a-a3d4-da19bba238ee" (UID: "ecc56888-0e26-441a-a3d4-da19bba238ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:10.353836 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.353754    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d006a24-6341-4d53-b847-e49793bd0531-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.353836 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.353802    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d006a24-6341-4d53-b847-e49793bd0531-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.353836 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.353820    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc56888-0e26-441a-a3d4-da19bba238ee-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.354037 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.353836    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqp2k\" (UniqueName: \"kubernetes.io/projected/6d006a24-6341-4d53-b847-e49793bd0531-kube-api-access-nqp2k\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.354037 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.353852    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8xw4\" (UniqueName: \"kubernetes.io/projected/ecc56888-0e26-441a-a3d4-da19bba238ee-kube-api-access-v8xw4\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.354037 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.353867    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc56888-0e26-441a-a3d4-da19bba238ee-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:10.481049 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.481013    4839 generic.go:356] "Generic (PLEG): container finished" podID="6d006a24-6341-4d53-b847-e49793bd0531" containerID="66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b" exitCode=0
Oct 29 22:59:10.481246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.481057    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"
Oct 29 22:59:10.481246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.481100    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" event={"ID":"6d006a24-6341-4d53-b847-e49793bd0531","Type":"ContainerDied","Data":"66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b"}
Oct 29 22:59:10.481246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.481140    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr" event={"ID":"6d006a24-6341-4d53-b847-e49793bd0531","Type":"ContainerDied","Data":"e00c4041287dcfd548a5d4fb25ebd5283f6a969a4322ab7779910d9c7ca93acf"}
Oct 29 22:59:10.481246 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.481175    4839 scope.go:122] "RemoveContainer" containerID="66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b"
Oct 29 22:59:10.482711 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.482686    4839 generic.go:356] "Generic (PLEG): container finished" podID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerID="aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572" exitCode=0
Oct 29 22:59:10.482846 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.482721    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"
Oct 29 22:59:10.482922 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.482900    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" event={"ID":"ecc56888-0e26-441a-a3d4-da19bba238ee","Type":"ContainerDied","Data":"aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572"}
Oct 29 22:59:10.483001 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.482939    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" event={"ID":"ecc56888-0e26-441a-a3d4-da19bba238ee","Type":"ContainerDied","Data":"c46a74e518148d540aafc7bf311aa94f399d2aafdb25e772f194544b25e77d35"}
Oct 29 22:59:10.493396 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.493365    4839 scope.go:122] "RemoveContainer" containerID="66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b"
Oct 29 22:59:10.493746 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:10.493721    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b\": container with ID starting with 66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b not found: ID does not exist" containerID="66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b"
Oct 29 22:59:10.493858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.493751    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b"} err="failed to get container status \"66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b\": rpc error: code = NotFound desc = could not find container \"66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b\": container with ID starting with 66dd738522db86539cc7e83fcb5cbb7bd1ba6f270635efc6f5ebb5fb66c6140b not found: ID does not exist"
Oct 29 22:59:10.493858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.493769    4839 scope.go:122] "RemoveContainer" containerID="aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572"
Oct 29 22:59:10.503400 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.503385    4839 scope.go:122] "RemoveContainer" containerID="aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572"
Oct 29 22:59:10.503657 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:10.503641    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572\": container with ID starting with aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572 not found: ID does not exist" containerID="aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572"
Oct 29 22:59:10.503706 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.503664    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572"} err="failed to get container status \"aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572\": rpc error: code = NotFound desc = could not find container \"aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572\": container with ID starting with aae99dcad316493807ec66f5ceca0404c773b7c0044a8a2894355b2acc888572 not found: ID does not exist"
Oct 29 22:59:10.543752 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.543723    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"]
Oct 29 22:59:10.548411 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.548384    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-2-w4-dd3ff3-kserve-6dfd6855d4-74pnr"]
Oct 29 22:59:10.584295 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.584258    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"]
Oct 29 22:59:10.588869 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.588843    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s"]
Oct 29 22:59:10.654583 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.654508    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"]
Oct 29 22:59:10.655089 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655074    4839 cpu_manager.go:414] "containerMap: removing container" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main"
Oct 29 22:59:10.655132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655090    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main"
Oct 29 22:59:10.655132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655103    4839 cpu_manager.go:414] "containerMap: removing container" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main"
Oct 29 22:59:10.655132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655108    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main"
Oct 29 22:59:10.655132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655127    4839 cpu_manager.go:414] "containerMap: removing container" podUID="8c0ae155-dfcf-4998-82ab-335d40659825" containerName="revoke-keys"
Oct 29 22:59:10.655132 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655132    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c0ae155-dfcf-4998-82ab-335d40659825" containerName="revoke-keys"
Oct 29 22:59:10.655332 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655250    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d006a24-6341-4d53-b847-e49793bd0531" containerName="main"
Oct 29 22:59:10.655332 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655262    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main"
Oct 29 22:59:10.655332 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.655271    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c0ae155-dfcf-4998-82ab-335d40659825" containerName="revoke-keys"
Oct 29 22:59:10.660516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.660489    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.694216 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.694174    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"]
Oct 29 22:59:10.759127 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759091    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759312 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759294    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759329    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759349    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvj2c\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-kube-api-access-tvj2c\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759446 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759433    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759486 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759462    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759486 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759478    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759563 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759497    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.759663 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.759640    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861111    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861315    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861353    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861372    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvj2c\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-kube-api-access-tvj2c\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861417    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861463    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861489    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861517    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861555    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.861650 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861603    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-data\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.862127 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861660    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-credential-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.862127 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.861862    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-socket\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.862127 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.862062    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-certs\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.862346 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.862307    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istiod-ca-cert\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.863863 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.863839    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-envoy\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.864044 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.864016    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-podinfo\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.885308 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.885267    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-token\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.885443 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.885420    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvj2c\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-kube-api-access-tvj2c\") pod \"e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") " pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:10.975282 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:10.975248    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:11.011455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.011420    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d006a24-6341-4d53-b847-e49793bd0531" path="/var/lib/kubelet/pods/6d006a24-6341-4d53-b847-e49793bd0531/volumes"
Oct 29 22:59:11.012126 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.012099    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" path="/var/lib/kubelet/pods/ecc56888-0e26-441a-a3d4-da19bba238ee/volumes"
Oct 29 22:59:11.180857 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.180817    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-dd3ff3/distinct-w4-dd3ff3-kserve-57f6cdbdd7-2ks5s" podUID="ecc56888-0e26-441a-a3d4-da19bba238ee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:11.292164 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:59:11.292119    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f5a414_18f6_4063_9e8a_62d7cc7913dd.slice/crio-7c08d113745c3d58bf8037b82d31399dda75dd99e9957f58e56f688de29acebf WatchSource:0}: Error finding container 7c08d113745c3d58bf8037b82d31399dda75dd99e9957f58e56f688de29acebf: Status 404 returned error can't find the container with id 7c08d113745c3d58bf8037b82d31399dda75dd99e9957f58e56f688de29acebf
Oct 29 22:59:11.294779 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.294752    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"]
Oct 29 22:59:11.294923 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.294896    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:59:11.294969 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.294957    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:59:11.295001 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.294988    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:59:11.500387 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.500304    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:11.518038 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.517988    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" event={"ID":"65f5a414-18f6-4063-9e8a-62d7cc7913dd","Type":"ContainerStarted","Data":"27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a"}
Oct 29 22:59:11.518038 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.518063    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" event={"ID":"65f5a414-18f6-4063-9e8a-62d7cc7913dd","Type":"ContainerStarted","Data":"7c08d113745c3d58bf8037b82d31399dda75dd99e9957f58e56f688de29acebf"}
Oct 29 22:59:11.568962 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.568847    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podStartSLOduration=1.568834099 podStartE2EDuration="1.568834099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:11.565828097 +0000 UTC m=+1008.668904382" watchObservedRunningTime="2026-10-29 22:59:11.568834099 +0000 UTC m=+1008.671910370"
Oct 29 22:59:11.638248 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.628760    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:11.975519 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:11.975427    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:12.602785 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:12.602743    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:12.967804 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:12.967702    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"]
Oct 29 22:59:13.002639 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.002600    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:59:13.003237 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.003201    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:59:13.013647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.013620    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"]
Oct 29 22:59:13.015446 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.015419    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.093173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.089509    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:59:13.093173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.089591    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:13.122501 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.122452    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9nws\" (UniqueName: \"kubernetes.io/projected/4a40cb3c-ae14-41de-a4eb-2caa9b391428-kube-api-access-s9nws\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.122667 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.122650    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.122729 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.122676    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.223864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.223782    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9nws\" (UniqueName: \"kubernetes.io/projected/4a40cb3c-ae14-41de-a4eb-2caa9b391428-kube-api-access-s9nws\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.224017 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.223922    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.224017 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.223946    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.224630 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.224603    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.226401 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.226381    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-xks-ca\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.245866 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.245838    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9nws\" (UniqueName: \"kubernetes.io/projected/4a40cb3c-ae14-41de-a4eb-2caa9b391428-kube-api-access-s9nws\") pod \"maas-api-revoke-sub-761cf924-mmsgw\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.327603 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.327544    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:13.334599 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.334560    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:13.826931 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:59:13.826888    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a40cb3c_ae14_41de_a4eb_2caa9b391428.slice/crio-3f9dd654432c4b36e03b863ec17ed35c41a32000aa91725e76d75fc151174444 WatchSource:0}: Error finding container 3f9dd654432c4b36e03b863ec17ed35c41a32000aa91725e76d75fc151174444: Status 404 returned error can't find the container with id 3f9dd654432c4b36e03b863ec17ed35c41a32000aa91725e76d75fc151174444
Oct 29 22:59:13.829972 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.829946    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"]
Oct 29 22:59:13.990362 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.990316    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:59:13.990563 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:13.990387    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:59:14.596315 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.596270    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw" event={"ID":"4a40cb3c-ae14-41de-a4eb-2caa9b391428","Type":"ContainerStarted","Data":"198759c8d6ec73d9b5531869b097dcbd2d1235d6ca69179e68f55bcfa845db69"}
Oct 29 22:59:14.596758 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.596345    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw" event={"ID":"4a40cb3c-ae14-41de-a4eb-2caa9b391428","Type":"ContainerStarted","Data":"3f9dd654432c4b36e03b863ec17ed35c41a32000aa91725e76d75fc151174444"}
Oct 29 22:59:14.639936 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.639884    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw" podStartSLOduration=2.639869658 podStartE2EDuration="2.639869658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:14.639242388 +0000 UTC m=+1011.742318660" watchObservedRunningTime="2026-10-29 22:59:14.639869658 +0000 UTC m=+1011.742945929"
Oct 29 22:59:14.778171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.768878    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"]
Oct 29 22:59:14.779235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.779190    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg" podUID="1cb0f1d1-f491-49d1-a74b-99db2d490997" containerName="maas-api" containerID="cri-o://4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b" gracePeriod=30
Oct 29 22:59:14.840399 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.840361    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:14.984181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.978580    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:59:14.984181 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:14.978645    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:59:15.501168 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.492548    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.501583 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.501556    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:59:15.521188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.515633    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cb0f1d1-f491-49d1-a74b-99db2d490997-kube-api-access-tvld2\" (UniqueName: \"kubernetes.io/projected/1cb0f1d1-f491-49d1-a74b-99db2d490997-kube-api-access-tvld2\") pod \"1cb0f1d1-f491-49d1-a74b-99db2d490997\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") "
Oct 29 22:59:15.521188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.515869    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-maas-api-tls\") pod \"1cb0f1d1-f491-49d1-a74b-99db2d490997\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") "
Oct 29 22:59:15.521188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.515921    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-metrics-certs\") pod \"1cb0f1d1-f491-49d1-a74b-99db2d490997\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") "
Oct 29 22:59:15.521188 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.516025    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cb0f1d1-f491-49d1-a74b-99db2d490997-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0f1d1-f491-49d1-a74b-99db2d490997-openshift-service-ca\") pod \"1cb0f1d1-f491-49d1-a74b-99db2d490997\" (UID: \"1cb0f1d1-f491-49d1-a74b-99db2d490997\") "
Oct 29 22:59:15.525565 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.522366    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cb0f1d1-f491-49d1-a74b-99db2d490997-openshift-service-ca" pod "1cb0f1d1-f491-49d1-a74b-99db2d490997" (UID: "1cb0f1d1-f491-49d1-a74b-99db2d490997"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:15.635088 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.628522    4839 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cb0f1d1-f491-49d1-a74b-99db2d490997-openshift-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:15.636261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.636009    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.817686    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb0f1d1-f491-49d1-a74b-99db2d490997-kube-api-access-tvld2" pod "1cb0f1d1-f491-49d1-a74b-99db2d490997" (UID: "1cb0f1d1-f491-49d1-a74b-99db2d490997"). InnerVolumeSpecName "kube-api-access-tvld2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.818069    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-metrics-certs" pod "1cb0f1d1-f491-49d1-a74b-99db2d490997" (UID: "1cb0f1d1-f491-49d1-a74b-99db2d490997"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.818346    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.819763    4839 generic.go:356] "Generic (PLEG): container finished" podID="1cb0f1d1-f491-49d1-a74b-99db2d490997" containerID="4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b" exitCode=0
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.819942    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg" event={"ID":"1cb0f1d1-f491-49d1-a74b-99db2d490997","Type":"ContainerDied","Data":"4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b"}
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.819989    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg" event={"ID":"1cb0f1d1-f491-49d1-a74b-99db2d490997","Type":"ContainerDied","Data":"3ede2ea3e834c984842de7b056127524743a29495a8dd33eb1098ebcfab49f43"}
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.820016    4839 scope.go:122] "RemoveContainer" containerID="4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b"
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.820375    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.827092    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-maas-api-tls" pod "1cb0f1d1-f491-49d1-a74b-99db2d490997" (UID: "1cb0f1d1-f491-49d1-a74b-99db2d490997"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.828593    4839 generic.go:356] "Generic (PLEG): container finished" podID="4a40cb3c-ae14-41de-a4eb-2caa9b391428" containerID="198759c8d6ec73d9b5531869b097dcbd2d1235d6ca69179e68f55bcfa845db69" exitCode=0
Oct 29 22:59:15.828867 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.828646    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw" event={"ID":"4a40cb3c-ae14-41de-a4eb-2caa9b391428","Type":"ContainerDied","Data":"198759c8d6ec73d9b5531869b097dcbd2d1235d6ca69179e68f55bcfa845db69"}
Oct 29 22:59:15.838678 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.838629    4839 scope.go:122] "RemoveContainer" containerID="4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b"
Oct 29 22:59:15.838978 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:15.838954    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b\": container with ID starting with 4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b not found: ID does not exist" containerID="4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b"
Oct 29 22:59:15.839074 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.838982    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b"} err="failed to get container status \"4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b\": rpc error: code = NotFound desc = could not find container \"4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b\": container with ID starting with 4a7738edaa20985462865aaddc119de4d0effd4fcc1f152e1f4e053237d7188b not found: ID does not exist"
Oct 29 22:59:15.852998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.852971    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvld2\" (UniqueName: \"kubernetes.io/projected/1cb0f1d1-f491-49d1-a74b-99db2d490997-kube-api-access-tvld2\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:15.852998 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.852998    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-maas-api-tls\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:15.853206 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:15.853007    4839 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cb0f1d1-f491-49d1-a74b-99db2d490997-metrics-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:16.029049 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.022358    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:16.029049 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.022494    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:59:16.029049 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.022538    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:59:16.101234 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.097499    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"]
Oct 29 22:59:16.114271 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.114142    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2bd3f07e-2zbrc"]
Oct 29 22:59:16.124088 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.124031    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:16.285579 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.279216    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"]
Oct 29 22:59:16.328167 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.313998    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2bd3f07e-65478944dc-vjxxg"]
Oct 29 22:59:16.713969 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.713934    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:16.981131 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.980547    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:59:16.981131 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:16.980608    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:59:17.124219 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.117919    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cb0f1d1-f491-49d1-a74b-99db2d490997" path="/var/lib/kubelet/pods/1cb0f1d1-f491-49d1-a74b-99db2d490997/volumes"
Oct 29 22:59:17.124219 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.118719    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c0ae155-dfcf-4998-82ab-335d40659825" path="/var/lib/kubelet/pods/8c0ae155-dfcf-4998-82ab-335d40659825/volumes"
Oct 29 22:59:17.377757 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.377727    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:17.635230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.632121    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-openshift-ca\") pod \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") "
Oct 29 22:59:17.635230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.632316    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-xks-ca\") pod \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") "
Oct 29 22:59:17.635230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.632448    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a40cb3c-ae14-41de-a4eb-2caa9b391428-kube-api-access-s9nws\" (UniqueName: \"kubernetes.io/projected/4a40cb3c-ae14-41de-a4eb-2caa9b391428-kube-api-access-s9nws\") pod \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\" (UID: \"4a40cb3c-ae14-41de-a4eb-2caa9b391428\") "
Oct 29 22:59:17.657406 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.657364    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-openshift-ca" pod "4a40cb3c-ae14-41de-a4eb-2caa9b391428" (UID: "4a40cb3c-ae14-41de-a4eb-2caa9b391428"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:17.751210 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.749394    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-xks-ca" pod "4a40cb3c-ae14-41de-a4eb-2caa9b391428" (UID: "4a40cb3c-ae14-41de-a4eb-2caa9b391428"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 22:59:17.769137 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.769074    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:17.769137 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.769107    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a40cb3c-ae14-41de-a4eb-2caa9b391428-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:17.792201 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.778928    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:17.848084 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.848046    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"]
Oct 29 22:59:17.870600 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.870559    4839 cpu_manager.go:414] "containerMap: removing container" podUID="4a40cb3c-ae14-41de-a4eb-2caa9b391428" containerName="revoke-keys"
Oct 29 22:59:17.870830 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.870806    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a40cb3c-ae14-41de-a4eb-2caa9b391428" containerName="revoke-keys"
Oct 29 22:59:17.871033 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.871015    4839 cpu_manager.go:414] "containerMap: removing container" podUID="1cb0f1d1-f491-49d1-a74b-99db2d490997" containerName="maas-api"
Oct 29 22:59:17.871164 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.871135    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cb0f1d1-f491-49d1-a74b-99db2d490997" containerName="maas-api"
Oct 29 22:59:17.871479 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.871458    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cb0f1d1-f491-49d1-a74b-99db2d490997" containerName="maas-api"
Oct 29 22:59:17.871616 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.871602    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a40cb3c-ae14-41de-a4eb-2caa9b391428" containerName="revoke-keys"
Oct 29 22:59:17.887807 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.887729    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.923343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.921570    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a40cb3c-ae14-41de-a4eb-2caa9b391428-kube-api-access-s9nws" pod "4a40cb3c-ae14-41de-a4eb-2caa9b391428" (UID: "4a40cb3c-ae14-41de-a4eb-2caa9b391428"). InnerVolumeSpecName "kube-api-access-s9nws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:17.923343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.921969    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-34d0fd42-openshift-default-dockercfg-m6vh4\""
Oct 29 22:59:17.998230 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.998193    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-envoy\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.998484 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.998463    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-socket\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.998609 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.998594    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-data\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.998743 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.998728    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596dcc57-b3bc-4de3-a806-d095be93c235-istio-podinfo\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.998841 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.998828    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596dcc57-b3bc-4de3-a806-d095be93c235-istiod-ca-cert\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.999000 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.998978    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4xbt\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-kube-api-access-g4xbt\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.999212 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.999194    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-certs\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.999364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.999348    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-istio-token\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.999474 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.999460    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-credential-socket\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:17.999837 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:17.999817    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9nws\" (UniqueName: \"kubernetes.io/projected/4a40cb3c-ae14-41de-a4eb-2caa9b391428-kube-api-access-s9nws\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:18.000688 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.000664    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:59:18.000878 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.000845    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:59:18.080967 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.080932    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"]
Oct 29 22:59:18.122178 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.112876    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-envoy\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.122486 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.122461    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-socket\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.122662 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.122643    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-data\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.122831 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.122812    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596dcc57-b3bc-4de3-a806-d095be93c235-istio-podinfo\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.123708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.122941    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596dcc57-b3bc-4de3-a806-d095be93c235-istiod-ca-cert\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.123708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.123033    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4xbt\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-kube-api-access-g4xbt\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.123708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.123214    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-certs\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.123708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.123256    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-istio-token\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.123708 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.123287    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-credential-socket\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.124641 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.124618    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-socket\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.143337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.125594    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-data\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.143337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.127938    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-certs\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.143337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.128463    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-credential-socket\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.143337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.131629    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596dcc57-b3bc-4de3-a806-d095be93c235-istiod-ca-cert\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.143337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.131802    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-envoy\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.143337 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.142910    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596dcc57-b3bc-4de3-a806-d095be93c235-istio-podinfo\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.145222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.145197    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"
Oct 29 22:59:18.145341 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.145263    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw" event={"ID":"4a40cb3c-ae14-41de-a4eb-2caa9b391428","Type":"ContainerDied","Data":"3f9dd654432c4b36e03b863ec17ed35c41a32000aa91725e76d75fc151174444"}
Oct 29 22:59:18.145414 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.145404    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f9dd654432c4b36e03b863ec17ed35c41a32000aa91725e76d75fc151174444"
Oct 29 22:59:18.149056 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.149033    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4xbt\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-kube-api-access-g4xbt\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.151314 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.151295    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:18.151415 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.151381    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-istio-token\") pod \"e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") " pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.394229 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.383987    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:18.461170 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.450578    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:18.986398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.984968    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 29 22:59:18.986398 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.985049    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 29 22:59:18.996489 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:18.996452    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"]
Oct 29 22:59:18.996904 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:59:18.996875    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596dcc57_b3bc_4de3_a806_d095be93c235.slice/crio-fd06bf9beaa0b86d55b5ae63d4926d08d83ea805886256af6c7fb857c1df3f97 WatchSource:0}: Error finding container fd06bf9beaa0b86d55b5ae63d4926d08d83ea805886256af6c7fb857c1df3f97: Status 404 returned error can't find the container with id fd06bf9beaa0b86d55b5ae63d4926d08d83ea805886256af6c7fb857c1df3f97
Oct 29 22:59:19.000090 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:19.000047    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:59:19.000244 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:19.000140    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:59:19.000244 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:19.000199    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 22:59:19.159357 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:19.159317    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" event={"ID":"596dcc57-b3bc-4de3-a806-d095be93c235","Type":"ContainerStarted","Data":"703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152"}
Oct 29 22:59:19.159525 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:19.159398    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" event={"ID":"596dcc57-b3bc-4de3-a806-d095be93c235","Type":"ContainerStarted","Data":"fd06bf9beaa0b86d55b5ae63d4926d08d83ea805886256af6c7fb857c1df3f97"}
Oct 29 22:59:19.210047 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:19.209926    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podStartSLOduration=2.209906075 podStartE2EDuration="2.209906075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:19.20562792 +0000 UTC m=+1016.308704192" watchObservedRunningTime="2026-10-29 22:59:19.209906075 +0000 UTC m=+1016.312982347"
Oct 29 22:59:19.463338 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:19.463300    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:20.162053 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:20.162025    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:20.162614 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:20.162558    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:20.163623 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:20.163598    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 22:59:20.296552 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:20.296511    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:20.487225 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:20.486302    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:20.814468 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:20.814431    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:20.921233 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:20.921197    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:21.175505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:21.175412    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:21.176250 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:21.176229    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 22:59:22.075243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:22.075203    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"]
Oct 29 22:59:22.079656 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:22.079628    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05eb199b-kdt75"]
Oct 29 22:59:22.089103 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:22.089077    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:22.635196 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:22.635128    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:22.754338 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:22.754308    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:22.981218 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:22.976270    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"]
Oct 29 22:59:23.203248 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.203219    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.234504 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.230496    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-34d0fd42\""
Oct 29 22:59:23.251367 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.251325    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtsw\" (UniqueName: \"kubernetes.io/projected/75f5a631-acbe-4e70-8ce9-0a89740c9694-kube-api-access-zrtsw\") pod \"payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") " pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.270265 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.270224    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5a631-acbe-4e70-8ce9-0a89740c9694-plugins-config-volume\") pod \"payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") " pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.288356 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.288300    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101cba55-58fa-4cf9-b98d-2a27e4fcd797" path="/var/lib/kubelet/pods/101cba55-58fa-4cf9-b98d-2a27e4fcd797/volumes"
Oct 29 22:59:23.289014 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.288992    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"]
Oct 29 22:59:23.303778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.302965    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-34d0fd42-dockercfg-ktxlp\""
Oct 29 22:59:23.371970 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.371929    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtsw\" (UniqueName: \"kubernetes.io/projected/75f5a631-acbe-4e70-8ce9-0a89740c9694-kube-api-access-zrtsw\") pod \"payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") " pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.372253 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.372225    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5a631-acbe-4e70-8ce9-0a89740c9694-plugins-config-volume\") pod \"payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") " pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.372847 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.372824    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5a631-acbe-4e70-8ce9-0a89740c9694-plugins-config-volume\") pod \"payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") " pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.394021 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.393988    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtsw\" (UniqueName: \"kubernetes.io/projected/75f5a631-acbe-4e70-8ce9-0a89740c9694-kube-api-access-zrtsw\") pod \"payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") " pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.617119 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.617080    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:23.920588 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:59:23.920556    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f5a631_acbe_4e70_8ce9_0a89740c9694.slice/crio-38e858b9456e8d4410ff996a172fdbde3d2520f64ac6dc09429e48bc4487a428 WatchSource:0}: Error finding container 38e858b9456e8d4410ff996a172fdbde3d2520f64ac6dc09429e48bc4487a428: Status 404 returned error can't find the container with id 38e858b9456e8d4410ff996a172fdbde3d2520f64ac6dc09429e48bc4487a428
Oct 29 22:59:23.924081 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:23.924053    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"]
Oct 29 22:59:24.061758 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.061724    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:24.313843 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.313811    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf" event={"ID":"75f5a631-acbe-4e70-8ce9-0a89740c9694","Type":"ContainerStarted","Data":"ec5022ab529535736db0b4807d40b218baaabf9049fa13077d723965ac0fc502"}
Oct 29 22:59:24.314029 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.313867    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf" event={"ID":"75f5a631-acbe-4e70-8ce9-0a89740c9694","Type":"ContainerStarted","Data":"38e858b9456e8d4410ff996a172fdbde3d2520f64ac6dc09429e48bc4487a428"}
Oct 29 22:59:24.314029 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.313951    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:24.380095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.380002    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf" podStartSLOduration=2.379955646 podStartE2EDuration="2.379955646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:24.379218726 +0000 UTC m=+1021.482294994" watchObservedRunningTime="2026-10-29 22:59:24.379955646 +0000 UTC m=+1021.483031958"
Oct 29 22:59:24.705500 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.705420    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"]
Oct 29 22:59:24.839100 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.839067    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:24.964218 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:24.959000    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:25.635169 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:25.629600    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:27.030079 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:27.030042    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"]
Oct 29 22:59:27.043731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:27.043698    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a294ddb4-4n7vv"]
Oct 29 22:59:27.274752 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:27.274722    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:28.618039 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:28.618008    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:29.040250 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.037065    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7262786-2553-4cd7-b798-143394491725" path="/var/lib/kubelet/pods/f7262786-2553-4cd7-b798-143394491725/volumes"
Oct 29 22:59:29.075172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.066936    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"]
Oct 29 22:59:29.077858 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.077805    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e6dfd5a-fzjnz"]
Oct 29 22:59:29.199249 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.191804    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:29.316893 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.316800    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:29.423183 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.378982    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"]
Oct 29 22:59:29.444024 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.428427    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:29.445389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.445360    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dd3ff3-z774m"]
Oct 29 22:59:29.706699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.706557    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw" podUID="27404b4f-4283-4e65-bd72-c366b6751331" containerName="payload-processing" containerID="cri-o://da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c" gracePeriod=25
Oct 29 22:59:29.833457 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.833425    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:29.977578 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:29.977509    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:59:30.049667 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.049633    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27404b4f-4283-4e65-bd72-c366b6751331-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27404b4f-4283-4e65-bd72-c366b6751331-plugins-config-volume\") pod \"27404b4f-4283-4e65-bd72-c366b6751331\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") "
Oct 29 22:59:30.049824 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.049772    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27404b4f-4283-4e65-bd72-c366b6751331-kube-api-access-ts99k\" (UniqueName: \"kubernetes.io/projected/27404b4f-4283-4e65-bd72-c366b6751331-kube-api-access-ts99k\") pod \"27404b4f-4283-4e65-bd72-c366b6751331\" (UID: \"27404b4f-4283-4e65-bd72-c366b6751331\") "
Oct 29 22:59:30.050016 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.049995    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27404b4f-4283-4e65-bd72-c366b6751331-plugins-config-volume" pod "27404b4f-4283-4e65-bd72-c366b6751331" (UID: "27404b4f-4283-4e65-bd72-c366b6751331"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:30.050375 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.050353    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27404b4f-4283-4e65-bd72-c366b6751331-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.052176 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.052126    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27404b4f-4283-4e65-bd72-c366b6751331-kube-api-access-ts99k" pod "27404b4f-4283-4e65-bd72-c366b6751331" (UID: "27404b4f-4283-4e65-bd72-c366b6751331"). InnerVolumeSpecName "kube-api-access-ts99k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:30.152282 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.152243    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts99k\" (UniqueName: \"kubernetes.io/projected/27404b4f-4283-4e65-bd72-c366b6751331-kube-api-access-ts99k\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:30.461476 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.461435    4839 generic.go:356] "Generic (PLEG): container finished" podID="27404b4f-4283-4e65-bd72-c366b6751331" containerID="da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c" exitCode=0
Oct 29 22:59:30.461625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.461485    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"
Oct 29 22:59:30.461625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.461489    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw" event={"ID":"27404b4f-4283-4e65-bd72-c366b6751331","Type":"ContainerDied","Data":"da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c"}
Oct 29 22:59:30.461625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.461543    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw" event={"ID":"27404b4f-4283-4e65-bd72-c366b6751331","Type":"ContainerDied","Data":"e208c6f524f0b2c2ecf798ff9d7bdd2ae5ea618910333fb7475a9775a12134ca"}
Oct 29 22:59:30.461625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.461573    4839 scope.go:122] "RemoveContainer" containerID="da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c"
Oct 29 22:59:30.477382 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.477354    4839 scope.go:122] "RemoveContainer" containerID="da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c"
Oct 29 22:59:30.477747 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:30.477703    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c\": container with ID starting with da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c not found: ID does not exist" containerID="da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c"
Oct 29 22:59:30.477864 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.477751    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c"} err="failed to get container status \"da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c\": rpc error: code = NotFound desc = could not find container \"da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c\": container with ID starting with da63593a289539f9f2161d17738874d3f22a6e03d1e40b434fd2d66d3694665c not found: ID does not exist"
Oct 29 22:59:30.524902 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.524873    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"]
Oct 29 22:59:30.538855 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:30.538823    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-01e33b3d-5566d76c89-8dtdw"]
Oct 29 22:59:31.011815 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:31.011785    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27404b4f-4283-4e65-bd72-c366b6751331" path="/var/lib/kubelet/pods/27404b4f-4283-4e65-bd72-c366b6751331/volumes"
Oct 29 22:59:31.012326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:31.012294    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71c997b-f5e5-4cdd-b015-bed43096e5ce" path="/var/lib/kubelet/pods/d71c997b-f5e5-4cdd-b015-bed43096e5ce/volumes"
Oct 29 22:59:31.012720 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:31.012706    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e5e531-c774-44bc-9dd5-d508f571bd7d" path="/var/lib/kubelet/pods/d8e5e531-c774-44bc-9dd5-d508f571bd7d/volumes"
Oct 29 22:59:31.167483 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:31.167450    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:32.986135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:32.985916    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.105594 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:33.105537    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.273485 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:33.270767    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.433171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:33.419443    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:33.663220 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:33.661664    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 22:59:34.857505 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:34.857473    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:34.977326 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:34.977051    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:35.177343 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:35.176963    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:35.449395 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:35.448708    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 22:59:36.495184 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:36.494348    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:36.609179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:36.609094    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:36.868346 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:36.868313    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:37.065784 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.065743    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:37.463173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.445717    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"]
Oct 29 22:59:37.463769 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.463702    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy" containerID="cri-o://be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2" gracePeriod=30
Oct 29 22:59:37.464052 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.455831    4839 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:37.553171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.541316    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"]
Oct 29 22:59:37.553171 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.541592    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy" containerID="cri-o://cb3c56cc7568a2220b810f578bae6379e3d3a6f061a3acc8f5209c96703c3eb6" gracePeriod=30
Oct 29 22:59:37.554656 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.554588    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded"
Oct 29 22:59:37.562935 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.562895    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:37.618172 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.607756    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:37.642179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.636744    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:37.666535 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.651360    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"]
Oct 29 22:59:37.666535 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.651609    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy" containerID="cri-o://1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf" gracePeriod=30
Oct 29 22:59:37.705179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.686682    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"]
Oct 29 22:59:37.705179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.687045    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy" containerID="cri-o://37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad" gracePeriod=30
Oct 29 22:59:37.705179 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:37.687524    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 22:59:38.716487 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:38.716436    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:39.264960 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:39.264928    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.050699 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:40.050646    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.646364 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:40.646319    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.808068 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:40.808033    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:40.929598 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:40.929495    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:41.036288 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:41.036255    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:41.675751 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:41.675718    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:42.028959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:42.028926    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"]
Oct 29 22:59:42.042703 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:42.042671    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-768356ed-bcvbv"]
Oct 29 22:59:42.158823 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:42.158790    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:42.468582 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:42.461584    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:43.000833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.000794    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:43.014103 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.014073    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c934a0b9-6681-4d93-830d-ba7fb7334a45" path="/var/lib/kubelet/pods/c934a0b9-6681-4d93-830d-ba7fb7334a45/volumes"
Oct 29 22:59:43.311980 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.311951    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:59:43.351732 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.351697    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:59:43.405363 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.405332    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:59:43.440789 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.440701    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-certs\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.440959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.440787    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-istio-token\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-istio-token\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.440959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.440829    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-envoy\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.440959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.440870    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ad91ce-c59e-4a72-9369-018e882bff72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ad91ce-c59e-4a72-9369-018e882bff72-istiod-ca-cert\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.440959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.440907    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-credential-socket\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.440959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.440949    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-certs" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.440959 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.440955    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-istio-token\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-istio-token\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.441331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441001    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-socket\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.441331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441067    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-data\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.441331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441128    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-kube-api-access-lgmrr\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-kube-api-access-lgmrr\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.441331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441185    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-credential-socket" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.441331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441235    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-socket\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.441331 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441290    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-envoy\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.441647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441342    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-certs\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.441647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441423    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52ad91ce-c59e-4a72-9369-018e882bff72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ad91ce-c59e-4a72-9369-018e882bff72-istio-podinfo\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.441647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441483    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d24105-1239-4048-9d39-35c906573fc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d24105-1239-4048-9d39-35c906573fc5-istiod-ca-cert\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.441647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441543    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5d24105-1239-4048-9d39-35c906573fc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d24105-1239-4048-9d39-35c906573fc5-istio-podinfo\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.441647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441572    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-kube-api-access-vmmpn\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-kube-api-access-vmmpn\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.441647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441606    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-data\") pod \"a5d24105-1239-4048-9d39-35c906573fc5\" (UID: \"a5d24105-1239-4048-9d39-35c906573fc5\") "
Oct 29 22:59:43.441647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441600    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-socket" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.441988 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.441638    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-credential-socket\") pod \"52ad91ce-c59e-4a72-9369-018e882bff72\" (UID: \"52ad91ce-c59e-4a72-9369-018e882bff72\") "
Oct 29 22:59:43.442305 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.442275    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ad91ce-c59e-4a72-9369-018e882bff72-istiod-ca-cert" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:43.443175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.442645    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d24105-1239-4048-9d39-35c906573fc5-istiod-ca-cert" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:43.443175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.443098    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52ad91ce-c59e-4a72-9369-018e882bff72-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.443175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.443124    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.443175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.443141    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.443175 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.443174    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.443511 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.443397    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-credential-socket" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.444826 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.444784    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-envoy" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.444950 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.444933    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-data" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.445095 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.444973    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5d24105-1239-4048-9d39-35c906573fc5-istio-podinfo" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:43.445510 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.445485    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-istio-token" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.446702 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.446677    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-istio-token" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.446803 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.446721    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-kube-api-access-lgmrr" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "kube-api-access-lgmrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.446889 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.446864    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-data" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.447078 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.447059    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52ad91ce-c59e-4a72-9369-018e882bff72-istio-podinfo" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:43.447183 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.447059    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-certs" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.447183 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.447079    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-socket" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.451267 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.447390    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-kube-api-access-vmmpn" pod "52ad91ce-c59e-4a72-9369-018e882bff72" (UID: "52ad91ce-c59e-4a72-9369-018e882bff72"). InnerVolumeSpecName "kube-api-access-vmmpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.459928 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.459900    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-envoy" pod "a5d24105-1239-4048-9d39-35c906573fc5" (UID: "a5d24105-1239-4048-9d39-35c906573fc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.544579 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.544545    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dfefde5-a0f8-41ce-ab36-39799b074499-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfefde5-a0f8-41ce-ab36-39799b074499-istiod-ca-cert\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.544778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.544630    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-credential-socket\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.544778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.544687    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-kube-api-access-qnm67\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-kube-api-access-qnm67\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.544778 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.544723    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-socket\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.544958 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.544888    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-data\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.545058 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545018    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-certs\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.545126 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545051    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-credential-socket" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.545126 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545114    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-token\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-token\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.545577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545121    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dfefde5-a0f8-41ce-ab36-39799b074499-istiod-ca-cert" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:43.545577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545243    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-socket" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.545577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545237    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-podinfo\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.545577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545271    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-data" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.545577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545366    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-certs" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.545577 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.545458    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-envoy\") pod \"8dfefde5-a0f8-41ce-ab36-39799b074499\" (UID: \"8dfefde5-a0f8-41ce-ab36-39799b074499\") "
Oct 29 22:59:43.546446 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546395    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d24105-1239-4048-9d39-35c906573fc5-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546446 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546426    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d24105-1239-4048-9d39-35c906573fc5-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546446 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546442    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmmpn\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-kube-api-access-vmmpn\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546457    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546480    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546495    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546507    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546519    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52ad91ce-c59e-4a72-9369-018e882bff72-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546532    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dfefde5-a0f8-41ce-ab36-39799b074499-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546545    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546556    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546569    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546582    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmrr\" (UniqueName: \"kubernetes.io/projected/a5d24105-1239-4048-9d39-35c906573fc5-kube-api-access-lgmrr\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546596    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546609    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546623    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d24105-1239-4048-9d39-35c906573fc5-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546635    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546649    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52ad91ce-c59e-4a72-9369-018e882bff72-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.546727 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.546662    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52ad91ce-c59e-4a72-9369-018e882bff72-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.548713 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.548453    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-kube-api-access-qnm67" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "kube-api-access-qnm67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.548713 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.548617    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-token" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.549222 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.549196    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-envoy" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.549766 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.549743    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-podinfo" pod "8dfefde5-a0f8-41ce-ab36-39799b074499" (UID: "8dfefde5-a0f8-41ce-ab36-39799b074499"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:43.569736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.569705    4839 generic.go:356] "Generic (PLEG): container finished" podID="52ad91ce-c59e-4a72-9369-018e882bff72" containerID="be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2" exitCode=0
Oct 29 22:59:43.569882 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.569768    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"
Oct 29 22:59:43.569882 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.569784    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" event={"ID":"52ad91ce-c59e-4a72-9369-018e882bff72","Type":"ContainerDied","Data":"be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2"}
Oct 29 22:59:43.569988 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.569887    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql" event={"ID":"52ad91ce-c59e-4a72-9369-018e882bff72","Type":"ContainerDied","Data":"f1d6b1634bae0e140654718c49d833aff117bcce37f6514d36efae63ff56e7fe"}
Oct 29 22:59:43.569988 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.569922    4839 scope.go:122] "RemoveContainer" containerID="be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2"
Oct 29 22:59:43.571625 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.571607    4839 generic.go:356] "Generic (PLEG): container finished" podID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerID="37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad" exitCode=0
Oct 29 22:59:43.571710 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.571690    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"
Oct 29 22:59:43.571710 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.571696    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" event={"ID":"8dfefde5-a0f8-41ce-ab36-39799b074499","Type":"ContainerDied","Data":"37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad"}
Oct 29 22:59:43.571805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.571741    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd" event={"ID":"8dfefde5-a0f8-41ce-ab36-39799b074499","Type":"ContainerDied","Data":"478d7eef837124d400709b6cd2eb99403ff6620cbf9a9cf3548a654fc84530b7"}
Oct 29 22:59:43.576370 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.576347    4839 generic.go:356] "Generic (PLEG): container finished" podID="a5d24105-1239-4048-9d39-35c906573fc5" containerID="1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf" exitCode=0
Oct 29 22:59:43.576492 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.576443    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"
Oct 29 22:59:43.576557 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.576442    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" event={"ID":"a5d24105-1239-4048-9d39-35c906573fc5","Type":"ContainerDied","Data":"1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf"}
Oct 29 22:59:43.576611 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.576586    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8" event={"ID":"a5d24105-1239-4048-9d39-35c906573fc5","Type":"ContainerDied","Data":"a54ae0a73b9f12a246b912c4183874d794da51ef539ca1ae0b6892a72e6cb051"}
Oct 29 22:59:43.578472 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.578448    4839 generic.go:356] "Generic (PLEG): container finished" podID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerID="cb3c56cc7568a2220b810f578bae6379e3d3a6f061a3acc8f5209c96703c3eb6" exitCode=0
Oct 29 22:59:43.578601 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.578523    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" event={"ID":"eff42ba4-c22f-4659-9dc9-cd039d8b569f","Type":"ContainerDied","Data":"cb3c56cc7568a2220b810f578bae6379e3d3a6f061a3acc8f5209c96703c3eb6"}
Oct 29 22:59:43.584805 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.584785    4839 scope.go:122] "RemoveContainer" containerID="020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7"
Oct 29 22:59:43.596972 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.596943    4839 scope.go:122] "RemoveContainer" containerID="be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2"
Oct 29 22:59:43.597297 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:43.597276    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2\": container with ID starting with be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2 not found: ID does not exist" containerID="be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2"
Oct 29 22:59:43.597393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.597315    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2"} err="failed to get container status \"be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2\": rpc error: code = NotFound desc = could not find container \"be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2\": container with ID starting with be8280f4535d146be1913f26f2db715eae6e32e3b283e9596586b6c64898b3c2 not found: ID does not exist"
Oct 29 22:59:43.597393 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.597333    4839 scope.go:122] "RemoveContainer" containerID="020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7"
Oct 29 22:59:43.597608 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:43.597590    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7\": container with ID starting with 020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7 not found: ID does not exist" containerID="020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7"
Oct 29 22:59:43.597666 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.597615    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7"} err="failed to get container status \"020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7\": rpc error: code = NotFound desc = could not find container \"020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7\": container with ID starting with 020715a45a9c5b3ba18b829fef4f276375deaccc75147be01ba25c61339857c7 not found: ID does not exist"
Oct 29 22:59:43.597666 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.597630    4839 scope.go:122] "RemoveContainer" containerID="37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad"
Oct 29 22:59:43.610640 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.610618    4839 scope.go:122] "RemoveContainer" containerID="37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad"
Oct 29 22:59:43.610962 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:43.610938    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad\": container with ID starting with 37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad not found: ID does not exist" containerID="37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad"
Oct 29 22:59:43.611074 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.610968    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad"} err="failed to get container status \"37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad\": rpc error: code = NotFound desc = could not find container \"37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad\": container with ID starting with 37e2b4bde58f13e84687b62e354d5d2d368c333d5145cd7005b484933714a0ad not found: ID does not exist"
Oct 29 22:59:43.611074 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.610987    4839 scope.go:122] "RemoveContainer" containerID="1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf"
Oct 29 22:59:43.621174 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.621140    4839 scope.go:122] "RemoveContainer" containerID="1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf"
Oct 29 22:59:43.621500 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:43.621481    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf\": container with ID starting with 1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf not found: ID does not exist" containerID="1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf"
Oct 29 22:59:43.621561 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.621507    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf"} err="failed to get container status \"1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf\": rpc error: code = NotFound desc = could not find container \"1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf\": container with ID starting with 1543ebcd2358cce89025d1c096e90c1b595dc564b9318782200346e9d0775cdf not found: ID does not exist"
Oct 29 22:59:43.631833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.631800    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"]
Oct 29 22:59:43.640512 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.640480    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-g9jw8"]
Oct 29 22:59:43.648627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.648539    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.648627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.648570    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.648627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.648586    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnm67\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-kube-api-access-qnm67\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.648627 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.648602    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dfefde5-a0f8-41ce-ab36-39799b074499-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.802649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.714333    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"]
Oct 29 22:59:43.802649 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.723101    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-cztrd"]
Oct 29 22:59:43.823947 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.816441    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"]
Oct 29 22:59:43.824278 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.823965    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-4dsql"]
Oct 29 22:59:43.874003 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.873976    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:59:43.889420 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889388    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-kube-api-access-d8mc9\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-kube-api-access-d8mc9\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889574 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889476    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-socket\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889574 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889518    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istiod-ca-cert\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889706 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889579    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-certs\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889706 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889641    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-credential-socket\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889706 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889690    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-data\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889729    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-envoy\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889763    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-socket" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.889965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889796    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-podinfo\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889856    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-token\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-token\") pod \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\" (UID: \"eff42ba4-c22f-4659-9dc9-cd039d8b569f\") "
Oct 29 22:59:43.889965 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889898    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istiod-ca-cert" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 22:59:43.890243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889946    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-data" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.890243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.889986    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-credential-socket" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.890243 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.890065    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-certs" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.890396 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.890368    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.890451 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.890387    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.890451 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.890435    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.890589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.890450    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.890589 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.890463    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.892137 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.892074    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-kube-api-access-d8mc9" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "kube-api-access-d8mc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.892385 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.892278    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-envoy" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 22:59:43.892385 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.892287    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-podinfo" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 22:59:43.892508 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.892419    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-token" pod "eff42ba4-c22f-4659-9dc9-cd039d8b569f" (UID: "eff42ba4-c22f-4659-9dc9-cd039d8b569f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 22:59:43.991539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.991499    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8mc9\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-kube-api-access-d8mc9\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.991539 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.991536    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.991802 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.991550    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:43.991802 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:43.991562    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff42ba4-c22f-4659-9dc9-cd039d8b569f-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 22:59:44.592647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:44.592616    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"
Oct 29 22:59:44.592647 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:44.592639    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2" event={"ID":"eff42ba4-c22f-4659-9dc9-cd039d8b569f","Type":"ContainerDied","Data":"b37ba5dae006e2d70eb2be79c8c042561732496441d7f10cde493d25e3c14b29"}
Oct 29 22:59:44.593140 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:44.592702    4839 scope.go:122] "RemoveContainer" containerID="cb3c56cc7568a2220b810f578bae6379e3d3a6f061a3acc8f5209c96703c3eb6"
Oct 29 22:59:44.682736 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:44.682697    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"]
Oct 29 22:59:44.711731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:44.711699    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dd3ff3-openshift-default-74ffbb64df-ls5c2"]
Oct 29 22:59:44.855347 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:44.855260    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:45.023951 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:45.023915    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" path="/var/lib/kubelet/pods/52ad91ce-c59e-4a72-9369-018e882bff72/volumes"
Oct 29 22:59:45.024510 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:45.024488    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" path="/var/lib/kubelet/pods/8dfefde5-a0f8-41ce-ab36-39799b074499/volumes"
Oct 29 22:59:45.024977 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:45.024961    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d24105-1239-4048-9d39-35c906573fc5" path="/var/lib/kubelet/pods/a5d24105-1239-4048-9d39-35c906573fc5/volumes"
Oct 29 22:59:45.025731 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:45.025716    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" path="/var/lib/kubelet/pods/eff42ba4-c22f-4659-9dc9-cd039d8b569f/volumes"
Oct 29 22:59:45.170531 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:45.170448    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:45.279958 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:45.279918    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:46.035799 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:46.035755    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"]
Oct 29 22:59:46.039932 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:46.039907    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"]
Oct 29 22:59:46.043748 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:46.043723    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c712ec65-lwnlb"]
Oct 29 22:59:46.047389 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:46.047366    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec018b88-ldmg9"]
Oct 29 22:59:46.702520 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:46.702481    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:46.945028 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:46.944993    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:47.010604 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:47.010518    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3763dc23-27ce-409e-b286-8e98c1083ee2" path="/var/lib/kubelet/pods/3763dc23-27ce-409e-b286-8e98c1083ee2/volumes"
Oct 29 22:59:47.011161 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:47.011128    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8214a394-2030-4c68-ac32-eba04c461ca2" path="/var/lib/kubelet/pods/8214a394-2030-4c68-ac32-eba04c461ca2/volumes"
Oct 29 22:59:47.240001 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:47.239970    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:48.601853 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:48.601825    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:50.368421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:50.368385    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:51.993205 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:51.992558    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:53.768135 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:53.768100    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:55.030516 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:55.030478    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"]
Oct 29 22:59:55.044261 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:55.044216    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4ab029a-lz5gb"]
Oct 29 22:59:55.376586 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:55.376487    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:57.010086 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.010051    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c78859-6b20-426d-91e1-4d968380fe6d" path="/var/lib/kubelet/pods/46c78859-6b20-426d-91e1-4d968380fe6d/volumes"
Oct 29 22:59:57.020134 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.020103    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:57.030045 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.030018    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"]
Oct 29 22:59:57.043482 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.043460    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ea6316-p7rzg"]
Oct 29 22:59:57.519833 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.519799    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"]
Oct 29 22:59:57.520365 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520351    4839 cpu_manager.go:414] "containerMap: removing container" podUID="27404b4f-4283-4e65-bd72-c366b6751331" containerName="payload-processing"
Oct 29 22:59:57.520421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520366    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27404b4f-4283-4e65-bd72-c366b6751331" containerName="payload-processing"
Oct 29 22:59:57.520421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520380    4839 cpu_manager.go:414] "containerMap: removing container" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy"
Oct 29 22:59:57.520421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520385    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy"
Oct 29 22:59:57.520421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520391    4839 cpu_manager.go:414] "containerMap: removing container" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy"
Oct 29 22:59:57.520421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520397    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy"
Oct 29 22:59:57.520421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520414    4839 cpu_manager.go:414] "containerMap: removing container" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy"
Oct 29 22:59:57.520421 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520419    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520425    4839 cpu_manager.go:414] "containerMap: removing container" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520430    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520448    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520452    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520530    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d24105-1239-4048-9d39-35c906573fc5" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520539    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dfefde5-a0f8-41ce-ab36-39799b074499" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520545    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="27404b4f-4283-4e65-bd72-c366b6751331" containerName="payload-processing"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520552    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520559    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff42ba4-c22f-4659-9dc9-cd039d8b569f" containerName="istio-proxy"
Oct 29 22:59:57.520635 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.520564    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ad91ce-c59e-4a72-9369-018e882bff72" containerName="istio-proxy"
Oct 29 22:59:57.526076 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.526051    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.542194 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.542163    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"]
Oct 29 22:59:57.543863 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.543838    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 22:59:57.548928 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.548901    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 22:59:57.600054 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.600010    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.600455 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.600423    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654gm\" (UniqueName: \"kubernetes.io/projected/39dfc716-e162-4571-ad9c-7af8f42a96b1-kube-api-access-654gm\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.600756 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.600731    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.701859 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.701826    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654gm\" (UniqueName: \"kubernetes.io/projected/39dfc716-e162-4571-ad9c-7af8f42a96b1-kube-api-access-654gm\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.702024 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.701997    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.702138 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.702121    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.702807 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.702788    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.704536 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.704515    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.724235 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.724207    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654gm\" (UniqueName: \"kubernetes.io/projected/39dfc716-e162-4571-ad9c-7af8f42a96b1-kube-api-access-654gm\") pod \"maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:57.890467 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:57.890371    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 22:59:58.199015 ip-10-0-136-187 kubenswrapper[4839]: W1029 22:59:58.198978    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39dfc716_e162_4571_ad9c_7af8f42a96b1.slice/crio-275cd2367dcba2781b2f2c4be117078e1a50ad7978727103be28ac147e33d405 WatchSource:0}: Error finding container 275cd2367dcba2781b2f2c4be117078e1a50ad7978727103be28ac147e33d405: Status 404 returned error can't find the container with id 275cd2367dcba2781b2f2c4be117078e1a50ad7978727103be28ac147e33d405
Oct 29 22:59:58.202941 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.202917    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"]
Oct 29 22:59:58.747442 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.747408    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"]
Oct 29 22:59:58.751710 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.751689    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.786296 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.786256    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"]
Oct 29 22:59:58.800187 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.799100    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-28ed1cea-serving-cert\""
Oct 29 22:59:58.802173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.801234    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb" event={"ID":"39dfc716-e162-4571-ad9c-7af8f42a96b1","Type":"ContainerStarted","Data":"4b232778c3cccdb752d5f435869d1ac081f062ebbe424d6db1d3e4deb965e4bd"}
Oct 29 22:59:58.802173 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.801300    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb" event={"ID":"39dfc716-e162-4571-ad9c-7af8f42a96b1","Type":"ContainerStarted","Data":"275cd2367dcba2781b2f2c4be117078e1a50ad7978727103be28ac147e33d405"}
Oct 29 22:59:58.816874 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.816846    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.817025 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.816885    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-metrics-certs\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.817098 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.817076    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cdcn\" (UniqueName: \"kubernetes.io/projected/0b123ccd-a354-4534-90ed-d1467b5dcc96-kube-api-access-7cdcn\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.817157 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.817126    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b123ccd-a354-4534-90ed-d1467b5dcc96-openshift-service-ca\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.835004 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.834967    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:58.901293 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.901170    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb" podStartSLOduration=1.90113289 podStartE2EDuration="1.90113289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 22:59:58.894696285 +0000 UTC m=+1055.997772579" watchObservedRunningTime="2026-10-29 22:59:58.90113289 +0000 UTC m=+1056.004209162"
Oct 29 22:59:58.918121 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.918081    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.918121 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.918124    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-metrics-certs\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.918363 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:58.918280    4839 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-serving-cert: secret "maas-api-e2e-derive-28ed1cea-serving-cert" not found
Oct 29 22:59:58.918363 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.918341    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cdcn\" (UniqueName: \"kubernetes.io/projected/0b123ccd-a354-4534-90ed-d1467b5dcc96-kube-api-access-7cdcn\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.918363 ip-10-0-136-187 kubenswrapper[4839]: E1029 22:59:58.918358    4839 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls podName:0b123ccd-a354-4534-90ed-d1467b5dcc96 nodeName:}" failed. No retries permitted until 2026-10-29 22:59:59.418336642 +0000 UTC m=+1056.521412909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls") pod "maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb" (UID: "0b123ccd-a354-4534-90ed-d1467b5dcc96") : secret "maas-api-e2e-derive-28ed1cea-serving-cert" not found
Oct 29 22:59:58.918531 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.918510    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b123ccd-a354-4534-90ed-d1467b5dcc96-openshift-service-ca\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.919104 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.919077    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b123ccd-a354-4534-90ed-d1467b5dcc96-openshift-service-ca\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.920804 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.920781    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-metrics-certs\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:58.951098 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.945007    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:58.999825 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:58.999785    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cdcn\" (UniqueName: \"kubernetes.io/projected/0b123ccd-a354-4534-90ed-d1467b5dcc96-kube-api-access-7cdcn\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:59.021613 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:59.021576    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c104c251-13d8-4bee-9888-5f3c381c1d29" path="/var/lib/kubelet/pods/c104c251-13d8-4bee-9888-5f3c381c1d29/volumes"
Oct 29 22:59:59.197935 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:59.197824    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 22:59:59.426631 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:59.426596    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:59.429005 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:59.428979    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls\") pod \"maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:59.668228 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:59.667872    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 22:59:59.823619 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:59.823449    4839 generic.go:356] "Generic (PLEG): container finished" podID="39dfc716-e162-4571-ad9c-7af8f42a96b1" containerID="4b232778c3cccdb752d5f435869d1ac081f062ebbe424d6db1d3e4deb965e4bd" exitCode=0
Oct 29 22:59:59.823945 ip-10-0-136-187 kubenswrapper[4839]: I1029 22:59:59.823673    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb" event={"ID":"39dfc716-e162-4571-ad9c-7af8f42a96b1","Type":"ContainerDied","Data":"4b232778c3cccdb752d5f435869d1ac081f062ebbe424d6db1d3e4deb965e4bd"}
Oct 29 23:00:00.065208 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.058081    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"]
Oct 29 23:00:00.223736 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.223694    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"]
Oct 29 23:00:00.364455 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.364377    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.382134 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.375439    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"]
Oct 29 23:00:00.431619 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.429466    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.432084 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.431894    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"]
Oct 29 23:00:00.439030 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.437804    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.439628 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.439532    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"]
Oct 29 23:00:00.445430 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.445399    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"]
Oct 29 23:00:00.450104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.449988    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"]
Oct 29 23:00:00.493553 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.493508    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j75kh\" (UniqueName: \"kubernetes.io/projected/9befbf1b-c732-4d0e-9a4f-2823c644240f-kube-api-access-j75kh\") pod \"maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.494306 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.494262    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4hb\" (UniqueName: \"kubernetes.io/projected/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-kube-api-access-vq4hb\") pod \"maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.494466 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.494448    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.494527 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.494487    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9befbf1b-c732-4d0e-9a4f-2823c644240f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.599076 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.599042    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9747478-025e-41ee-84cd-ea0e11bdc24c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.599294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.599117    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j75kh\" (UniqueName: \"kubernetes.io/projected/9befbf1b-c732-4d0e-9a4f-2823c644240f-kube-api-access-j75kh\") pod \"maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.599294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.599238    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4hb\" (UniqueName: \"kubernetes.io/projected/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-kube-api-access-vq4hb\") pod \"maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.599428 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.599375    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.599428 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.599406    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9befbf1b-c732-4d0e-9a4f-2823c644240f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.599557 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.599503    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkg5n\" (UniqueName: \"kubernetes.io/projected/b9747478-025e-41ee-84cd-ea0e11bdc24c-kube-api-access-zkg5n\") pod \"maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.600184 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.600161    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9befbf1b-c732-4d0e-9a4f-2823c644240f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.600250 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.600184    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.621661 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.621580    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j75kh\" (UniqueName: \"kubernetes.io/projected/9befbf1b-c732-4d0e-9a4f-2823c644240f-kube-api-access-j75kh\") pod \"maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.621785 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.621671    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4hb\" (UniqueName: \"kubernetes.io/projected/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-kube-api-access-vq4hb\") pod \"maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.701187 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.701137    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkg5n\" (UniqueName: \"kubernetes.io/projected/b9747478-025e-41ee-84cd-ea0e11bdc24c-kube-api-access-zkg5n\") pod \"maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.701359 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.701278    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9747478-025e-41ee-84cd-ea0e11bdc24c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.701918 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.701896    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9747478-025e-41ee-84cd-ea0e11bdc24c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.714287 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.714254    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:00.725939 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.725903    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkg5n\" (UniqueName: \"kubernetes.io/projected/b9747478-025e-41ee-84cd-ea0e11bdc24c-kube-api-access-zkg5n\") pod \"maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.744471 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.744446    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:00.786036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.762121    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:00.786036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.775448    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:00.938173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:00.935456    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb" event={"ID":"0b123ccd-a354-4534-90ed-d1467b5dcc96","Type":"ContainerStarted","Data":"855ef0fa8db5d72a021015801ccfa326f83bbe3761fa9a5283c06d382dbf9690"}
Oct 29 23:00:01.257786 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.257751    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 23:00:01.269120 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.269070    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"]
Oct 29 23:00:01.314002 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.313971    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"]
Oct 29 23:00:01.322379 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:01.322311    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9747478_025e_41ee_84cd_ea0e11bdc24c.slice/crio-af1ca5bf0d0efdda07367a07508c566ed2fa0d82b86fa54f2a83dd38bb42e223 WatchSource:0}: Error finding container af1ca5bf0d0efdda07367a07508c566ed2fa0d82b86fa54f2a83dd38bb42e223: Status 404 returned error can't find the container with id af1ca5bf0d0efdda07367a07508c566ed2fa0d82b86fa54f2a83dd38bb42e223
Oct 29 23:00:01.324578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.323558    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dfc716-e162-4571-ad9c-7af8f42a96b1-kube-api-access-654gm\" (UniqueName: \"kubernetes.io/projected/39dfc716-e162-4571-ad9c-7af8f42a96b1-kube-api-access-654gm\") pod \"39dfc716-e162-4571-ad9c-7af8f42a96b1\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") "
Oct 29 23:00:01.324578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.323640    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-xks-ca\") pod \"39dfc716-e162-4571-ad9c-7af8f42a96b1\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") "
Oct 29 23:00:01.324578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.323809    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-openshift-ca\") pod \"39dfc716-e162-4571-ad9c-7af8f42a96b1\" (UID: \"39dfc716-e162-4571-ad9c-7af8f42a96b1\") "
Oct 29 23:00:01.325249 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.324935    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-openshift-ca" pod "39dfc716-e162-4571-ad9c-7af8f42a96b1" (UID: "39dfc716-e162-4571-ad9c-7af8f42a96b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:01.328482 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.328435    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dfc716-e162-4571-ad9c-7af8f42a96b1-kube-api-access-654gm" pod "39dfc716-e162-4571-ad9c-7af8f42a96b1" (UID: "39dfc716-e162-4571-ad9c-7af8f42a96b1"). InnerVolumeSpecName "kube-api-access-654gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:01.330556 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.330509    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-xks-ca" pod "39dfc716-e162-4571-ad9c-7af8f42a96b1" (UID: "39dfc716-e162-4571-ad9c-7af8f42a96b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:01.333046 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:01.333015    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9befbf1b_c732_4d0e_9a4f_2823c644240f.slice/crio-e70c01b0b74bf27e45c053559e500bdaadf9bcfadd42ab8fad5de78829a0b5c7 WatchSource:0}: Error finding container e70c01b0b74bf27e45c053559e500bdaadf9bcfadd42ab8fad5de78829a0b5c7: Status 404 returned error can't find the container with id e70c01b0b74bf27e45c053559e500bdaadf9bcfadd42ab8fad5de78829a0b5c7
Oct 29 23:00:01.346256 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.345715    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"]
Oct 29 23:00:01.425750 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.425719    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654gm\" (UniqueName: \"kubernetes.io/projected/39dfc716-e162-4571-ad9c-7af8f42a96b1-kube-api-access-654gm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:01.425750 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.425750    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:01.425888 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.425760    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dfc716-e162-4571-ad9c-7af8f42a96b1-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:01.944529 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.944486    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" event={"ID":"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c","Type":"ContainerStarted","Data":"286d24feec37e2495303186a3f750b859164cc97cbe60438f446c5c6aaddf117"}
Oct 29 23:00:01.945101 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.944546    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" event={"ID":"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c","Type":"ContainerStarted","Data":"cf559a43028fb7fe0b819d85025a27100780d7ba7c9a523821a3d3967c2231ae"}
Oct 29 23:00:01.946451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.946419    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb" event={"ID":"0b123ccd-a354-4534-90ed-d1467b5dcc96","Type":"ContainerStarted","Data":"6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630"}
Oct 29 23:00:01.946600 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.946484    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 23:00:01.947757 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.947732    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" event={"ID":"9befbf1b-c732-4d0e-9a4f-2823c644240f","Type":"ContainerStarted","Data":"e70c01b0b74bf27e45c053559e500bdaadf9bcfadd42ab8fad5de78829a0b5c7"}
Oct 29 23:00:01.949443 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.949404    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" event={"ID":"b9747478-025e-41ee-84cd-ea0e11bdc24c","Type":"ContainerStarted","Data":"bad1bc507bc488facb6366a9ee79b8c61a0806f3f0157f793f5a31af96d09aaa"}
Oct 29 23:00:01.949568 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.949456    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" event={"ID":"b9747478-025e-41ee-84cd-ea0e11bdc24c","Type":"ContainerStarted","Data":"af1ca5bf0d0efdda07367a07508c566ed2fa0d82b86fa54f2a83dd38bb42e223"}
Oct 29 23:00:01.950797 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.950778    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"
Oct 29 23:00:01.950907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.950782    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb" event={"ID":"39dfc716-e162-4571-ad9c-7af8f42a96b1","Type":"ContainerDied","Data":"275cd2367dcba2781b2f2c4be117078e1a50ad7978727103be28ac147e33d405"}
Oct 29 23:00:01.950958 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.950902    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="275cd2367dcba2781b2f2c4be117078e1a50ad7978727103be28ac147e33d405"
Oct 29 23:00:01.995299 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:01.995180    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" podStartSLOduration=1.99516099 podStartE2EDuration="1.99516099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:01.993387383 +0000 UTC m=+1059.096463656" watchObservedRunningTime="2026-10-29 23:00:01.99516099 +0000 UTC m=+1059.098237256"
Oct 29 23:00:02.044589 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.044540    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb" podStartSLOduration=3.46697967 podStartE2EDuration="4.044524996s" totalImagesPullingTime="577.545326ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 22:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.042391289 +0000 UTC m=+1059.145467559" watchObservedRunningTime="2026-10-29 23:00:02.044524996 +0000 UTC m=+1059.147601267"
Oct 29 23:00:02.115702 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.115636    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" podStartSLOduration=2.115617932 podStartE2EDuration="2.115617932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:02.113247333 +0000 UTC m=+1059.216323603" watchObservedRunningTime="2026-10-29 23:00:02.115617932 +0000 UTC m=+1059.218694204"
Oct 29 23:00:02.399173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.398664    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:02.960606 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.960519    4839 generic.go:356] "Generic (PLEG): container finished" podID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerID="6b33380b909ebb15b21f7c6f4186bfa14222cecd0e5004fddaea5f5e38a8abb0" exitCode=60
Oct 29 23:00:02.960993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.960608    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" event={"ID":"9befbf1b-c732-4d0e-9a4f-2823c644240f","Type":"ContainerDied","Data":"6b33380b909ebb15b21f7c6f4186bfa14222cecd0e5004fddaea5f5e38a8abb0"}
Oct 29 23:00:02.960993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.960893    4839 scope.go:122] "RemoveContainer" containerID="6b33380b909ebb15b21f7c6f4186bfa14222cecd0e5004fddaea5f5e38a8abb0"
Oct 29 23:00:02.962573 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.962548    4839 generic.go:356] "Generic (PLEG): container finished" podID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerID="bad1bc507bc488facb6366a9ee79b8c61a0806f3f0157f793f5a31af96d09aaa" exitCode=7
Oct 29 23:00:02.962722 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.962619    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" event={"ID":"b9747478-025e-41ee-84cd-ea0e11bdc24c","Type":"ContainerDied","Data":"bad1bc507bc488facb6366a9ee79b8c61a0806f3f0157f793f5a31af96d09aaa"}
Oct 29 23:00:02.962857 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.962842    4839 scope.go:122] "RemoveContainer" containerID="bad1bc507bc488facb6366a9ee79b8c61a0806f3f0157f793f5a31af96d09aaa"
Oct 29 23:00:02.964691 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.964466    4839 generic.go:356] "Generic (PLEG): container finished" podID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerID="286d24feec37e2495303186a3f750b859164cc97cbe60438f446c5c6aaddf117" exitCode=60
Oct 29 23:00:02.964691 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.964534    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" event={"ID":"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c","Type":"ContainerDied","Data":"286d24feec37e2495303186a3f750b859164cc97cbe60438f446c5c6aaddf117"}
Oct 29 23:00:02.964801 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:02.964704    4839 scope.go:122] "RemoveContainer" containerID="286d24feec37e2495303186a3f750b859164cc97cbe60438f446c5c6aaddf117"
Oct 29 23:00:03.240440 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:03.240332    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:00:03.976864 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.976828    4839 generic.go:356] "Generic (PLEG): container finished" podID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerID="10038dc0ac45bde46e5be542c4431055e882a4eb32f256f324bc69dd1375ec65" exitCode=60
Oct 29 23:00:03.977297 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.976911    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" event={"ID":"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c","Type":"ContainerDied","Data":"10038dc0ac45bde46e5be542c4431055e882a4eb32f256f324bc69dd1375ec65"}
Oct 29 23:00:03.977297 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.977016    4839 scope.go:122] "RemoveContainer" containerID="286d24feec37e2495303186a3f750b859164cc97cbe60438f446c5c6aaddf117"
Oct 29 23:00:03.977448 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.977430    4839 scope.go:122] "RemoveContainer" containerID="10038dc0ac45bde46e5be542c4431055e882a4eb32f256f324bc69dd1375ec65"
Oct 29 23:00:03.977896 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:03.977872    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn_odh-ai-gateway-infra(c69abd2b-cd07-40b8-a6d8-c4b2903bc80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c"
Oct 29 23:00:03.981492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.981457    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" event={"ID":"9befbf1b-c732-4d0e-9a4f-2823c644240f","Type":"ContainerStarted","Data":"8b220e460dad470e2847a95599afb0d678028337a887a049188c30a9231b2379"}
Oct 29 23:00:03.984458 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.984434    4839 generic.go:356] "Generic (PLEG): container finished" podID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerID="dbde8f0f2dd553cce982d0d32d9eaae742bd7a9e65bd9c779ddb98fbbcbd3f6e" exitCode=7
Oct 29 23:00:03.984584 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.984495    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" event={"ID":"b9747478-025e-41ee-84cd-ea0e11bdc24c","Type":"ContainerDied","Data":"dbde8f0f2dd553cce982d0d32d9eaae742bd7a9e65bd9c779ddb98fbbcbd3f6e"}
Oct 29 23:00:03.984843 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:03.984823    4839 scope.go:122] "RemoveContainer" containerID="dbde8f0f2dd553cce982d0d32d9eaae742bd7a9e65bd9c779ddb98fbbcbd3f6e"
Oct 29 23:00:03.985203 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:03.985182    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st_odh-ai-gateway-infra(b9747478-025e-41ee-84cd-ea0e11bdc24c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c"
Oct 29 23:00:04.028834 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.028796    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:04.032409 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.032387    4839 scope.go:122] "RemoveContainer" containerID="bad1bc507bc488facb6366a9ee79b8c61a0806f3f0157f793f5a31af96d09aaa"
Oct 29 23:00:04.166635 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.166569    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" podStartSLOduration=4.166547787 podStartE2EDuration="4.166547787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:04.135664942 +0000 UTC m=+1061.238741214" watchObservedRunningTime="2026-10-29 23:00:04.166547787 +0000 UTC m=+1061.269624059"
Oct 29 23:00:04.996740 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.996702    4839 scope.go:122] "RemoveContainer" containerID="10038dc0ac45bde46e5be542c4431055e882a4eb32f256f324bc69dd1375ec65"
Oct 29 23:00:04.997214 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:04.997080    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn_odh-ai-gateway-infra(c69abd2b-cd07-40b8-a6d8-c4b2903bc80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c"
Oct 29 23:00:04.998952 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.998927    4839 generic.go:356] "Generic (PLEG): container finished" podID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerID="8b220e460dad470e2847a95599afb0d678028337a887a049188c30a9231b2379" exitCode=60
Oct 29 23:00:04.999091 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.999007    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" event={"ID":"9befbf1b-c732-4d0e-9a4f-2823c644240f","Type":"ContainerDied","Data":"8b220e460dad470e2847a95599afb0d678028337a887a049188c30a9231b2379"}
Oct 29 23:00:04.999091 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.999082    4839 scope.go:122] "RemoveContainer" containerID="6b33380b909ebb15b21f7c6f4186bfa14222cecd0e5004fddaea5f5e38a8abb0"
Oct 29 23:00:04.999323 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:04.999307    4839 scope.go:122] "RemoveContainer" containerID="8b220e460dad470e2847a95599afb0d678028337a887a049188c30a9231b2379"
Oct 29 23:00:04.999644 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:04.999624    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r_odh-ai-gateway-infra(9befbf1b-c732-4d0e-9a4f-2823c644240f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f"
Oct 29 23:00:05.001276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:05.001258    4839 scope.go:122] "RemoveContainer" containerID="dbde8f0f2dd553cce982d0d32d9eaae742bd7a9e65bd9c779ddb98fbbcbd3f6e"
Oct 29 23:00:05.001640 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:05.001615    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st_odh-ai-gateway-infra(b9747478-025e-41ee-84cd-ea0e11bdc24c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c"
Oct 29 23:00:05.613065 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:05.613028    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:06.013266 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:06.013192    4839 scope.go:122] "RemoveContainer" containerID="8b220e460dad470e2847a95599afb0d678028337a887a049188c30a9231b2379"
Oct 29 23:00:06.013634 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:06.013470    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r_odh-ai-gateway-infra(9befbf1b-c732-4d0e-9a4f-2823c644240f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f"
Oct 29 23:00:06.661000 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:06.659614    4839 patch_prober.go:29] interesting pod/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:48968->10.133.0.100:15021: read: connection reset by peer" start-of-body=
Oct 29 23:00:06.661000 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:06.659674    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:48968->10.133.0.100:15021: read: connection reset by peer"
Oct 29 23:00:07.023427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.023392    4839 generic.go:356] "Generic (PLEG): container finished" podID="596dcc57-b3bc-4de3-a806-d095be93c235" containerID="703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152" exitCode=0
Oct 29 23:00:07.023824 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.023445    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" event={"ID":"596dcc57-b3bc-4de3-a806-d095be93c235","Type":"ContainerDied","Data":"703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152"}
Oct 29 23:00:07.023900 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.023840    4839 scope.go:122] "RemoveContainer" containerID="703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152"
Oct 29 23:00:07.025349 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.025322    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 23:00:07.025422 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.025391    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 23:00:07.025460 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.025423    4839 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 29 23:00:07.263179 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.263129    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"]
Oct 29 23:00:07.263735 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.263719    4839 cpu_manager.go:414] "containerMap: removing container" podUID="39dfc716-e162-4571-ad9c-7af8f42a96b1" containerName="revoke-keys"
Oct 29 23:00:07.263735 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.263733    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39dfc716-e162-4571-ad9c-7af8f42a96b1" containerName="revoke-keys"
Oct 29 23:00:07.263869 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.263854    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="39dfc716-e162-4571-ad9c-7af8f42a96b1" containerName="revoke-keys"
Oct 29 23:00:07.267492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.267472    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.305177 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.304895    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:07.389178 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.387022    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"]
Oct 29 23:00:07.418503 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.418117    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.418503 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.418233    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.418503 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.418316    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p56b5\" (UniqueName: \"kubernetes.io/projected/b1149292-e0a6-4f28-90c3-5147775bf9ba-kube-api-access-p56b5\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.425501 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.425465    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:07.519200 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.519158    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.519491 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.519231    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.519491 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.519311    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p56b5\" (UniqueName: \"kubernetes.io/projected/b1149292-e0a6-4f28-90c3-5147775bf9ba-kube-api-access-p56b5\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.519915 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.519888    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.522410 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.522387    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.541554 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.541490    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p56b5\" (UniqueName: \"kubernetes.io/projected/b1149292-e0a6-4f28-90c3-5147775bf9ba-kube-api-access-p56b5\") pod \"maas-api-revoke-sub-c3884fae-xbqm2\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.604743 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.604693    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:07.907205 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:07.907134    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1149292_e0a6_4f28_90c3_5147775bf9ba.slice/crio-cadbf41e228d36f508b6fd4c9dab7e613d457999a83848d62e6fb2594cb63505 WatchSource:0}: Error finding container cadbf41e228d36f508b6fd4c9dab7e613d457999a83848d62e6fb2594cb63505: Status 404 returned error can't find the container with id cadbf41e228d36f508b6fd4c9dab7e613d457999a83848d62e6fb2594cb63505
Oct 29 23:00:07.909260 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.909231    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"]
Oct 29 23:00:07.949158 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.949116    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"]
Oct 29 23:00:07.993996 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:07.993961    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 23:00:08.041790 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.041697    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" event={"ID":"596dcc57-b3bc-4de3-a806-d095be93c235","Type":"ContainerStarted","Data":"861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0"}
Oct 29 23:00:08.047100 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.045491    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"]
Oct 29 23:00:08.047100 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.046046    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2" event={"ID":"b1149292-e0a6-4f28-90c3-5147775bf9ba","Type":"ContainerStarted","Data":"cadbf41e228d36f508b6fd4c9dab7e613d457999a83848d62e6fb2594cb63505"}
Oct 29 23:00:08.063189 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.061214    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70b7884-srl99"]
Oct 29 23:00:08.106351 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.106312    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:08.256221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.255290    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:08.453479 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.453444    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 23:00:08.453739 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.453717    4839 patch_prober.go:29] interesting pod/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 23:00:08.453786 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:08.453769    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 23:00:09.010695 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:09.010663    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1972eab-70dc-45f5-8f92-3061b0d862c4" path="/var/lib/kubelet/pods/b1972eab-70dc-45f5-8f92-3061b0d862c4/volumes"
Oct 29 23:00:09.061490 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:09.061462    4839 generic.go:356] "Generic (PLEG): container finished" podID="b1149292-e0a6-4f28-90c3-5147775bf9ba" containerID="0d334f0dc86f461aa2ec786b6e2e5e37d584b184128c28fbaee535f90dd3aef4" exitCode=0
Oct 29 23:00:09.061871 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:09.061552    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2" event={"ID":"b1149292-e0a6-4f28-90c3-5147775bf9ba","Type":"ContainerDied","Data":"0d334f0dc86f461aa2ec786b6e2e5e37d584b184128c28fbaee535f90dd3aef4"}
Oct 29 23:00:09.097662 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:09.097613    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:09.453882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:09.453843    4839 patch_prober.go:29] interesting pod/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 23:00:09.454053 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:09.453902    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 23:00:10.465191 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.460107    4839 patch_prober.go:29] interesting pod/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 23:00:10.465191 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.460183    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 23:00:10.537449 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.537416    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:10.628442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.628413    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-openshift-ca\") pod \"b1149292-e0a6-4f28-90c3-5147775bf9ba\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") "
Oct 29 23:00:10.628619 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.628520    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1149292-e0a6-4f28-90c3-5147775bf9ba-kube-api-access-p56b5\" (UniqueName: \"kubernetes.io/projected/b1149292-e0a6-4f28-90c3-5147775bf9ba-kube-api-access-p56b5\") pod \"b1149292-e0a6-4f28-90c3-5147775bf9ba\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") "
Oct 29 23:00:10.628619 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.628561    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-xks-ca\") pod \"b1149292-e0a6-4f28-90c3-5147775bf9ba\" (UID: \"b1149292-e0a6-4f28-90c3-5147775bf9ba\") "
Oct 29 23:00:10.628873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.628845    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-openshift-ca" pod "b1149292-e0a6-4f28-90c3-5147775bf9ba" (UID: "b1149292-e0a6-4f28-90c3-5147775bf9ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:10.629418 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.629390    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:10.630846 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.630815    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-xks-ca" pod "b1149292-e0a6-4f28-90c3-5147775bf9ba" (UID: "b1149292-e0a6-4f28-90c3-5147775bf9ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:10.630947 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.630892    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1149292-e0a6-4f28-90c3-5147775bf9ba-kube-api-access-p56b5" pod "b1149292-e0a6-4f28-90c3-5147775bf9ba" (UID: "b1149292-e0a6-4f28-90c3-5147775bf9ba"). InnerVolumeSpecName "kube-api-access-p56b5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:10.730912 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.730819    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p56b5\" (UniqueName: \"kubernetes.io/projected/b1149292-e0a6-4f28-90c3-5147775bf9ba-kube-api-access-p56b5\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:10.730912 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.730852    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1149292-e0a6-4f28-90c3-5147775bf9ba-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:10.792171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:10.791820    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:11.102225 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:11.102182    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2" event={"ID":"b1149292-e0a6-4f28-90c3-5147775bf9ba","Type":"ContainerDied","Data":"cadbf41e228d36f508b6fd4c9dab7e613d457999a83848d62e6fb2594cb63505"}
Oct 29 23:00:11.102404 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:11.102257    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cadbf41e228d36f508b6fd4c9dab7e613d457999a83848d62e6fb2594cb63505"
Oct 29 23:00:11.102404 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:11.102197    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"
Oct 29 23:00:11.454170 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:11.454067    4839 patch_prober.go:29] interesting pod/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 23:00:11.454170 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:11.454122    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 23:00:12.414991 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:12.414950    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:12.454299 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:12.454266    4839 patch_prober.go:29] interesting pod/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 23:00:12.454451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:12.454317    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 23:00:12.950780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:12.950739    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf" podUID="75f5a631-acbe-4e70-8ce9-0a89740c9694" containerName="payload-processing" containerID="cri-o://ec5022ab529535736db0b4807d40b218baaabf9049fa13077d723965ac0fc502" gracePeriod=25
Oct 29 23:00:13.119387 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.119355    4839 generic.go:356] "Generic (PLEG): container finished" podID="75f5a631-acbe-4e70-8ce9-0a89740c9694" containerID="ec5022ab529535736db0b4807d40b218baaabf9049fa13077d723965ac0fc502" exitCode=0
Oct 29 23:00:13.119580 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.119424    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf" event={"ID":"75f5a631-acbe-4e70-8ce9-0a89740c9694","Type":"ContainerDied","Data":"ec5022ab529535736db0b4807d40b218baaabf9049fa13077d723965ac0fc502"}
Oct 29 23:00:13.303746 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.303719    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 23:00:13.396432 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.396387    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f5a631-acbe-4e70-8ce9-0a89740c9694-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5a631-acbe-4e70-8ce9-0a89740c9694-plugins-config-volume\") pod \"75f5a631-acbe-4e70-8ce9-0a89740c9694\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") "
Oct 29 23:00:13.396637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.396469    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f5a631-acbe-4e70-8ce9-0a89740c9694-kube-api-access-zrtsw\" (UniqueName: \"kubernetes.io/projected/75f5a631-acbe-4e70-8ce9-0a89740c9694-kube-api-access-zrtsw\") pod \"75f5a631-acbe-4e70-8ce9-0a89740c9694\" (UID: \"75f5a631-acbe-4e70-8ce9-0a89740c9694\") "
Oct 29 23:00:13.397914 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.397882    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f5a631-acbe-4e70-8ce9-0a89740c9694-plugins-config-volume" pod "75f5a631-acbe-4e70-8ce9-0a89740c9694" (UID: "75f5a631-acbe-4e70-8ce9-0a89740c9694"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:13.400821 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.400790    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f5a631-acbe-4e70-8ce9-0a89740c9694-kube-api-access-zrtsw" pod "75f5a631-acbe-4e70-8ce9-0a89740c9694" (UID: "75f5a631-acbe-4e70-8ce9-0a89740c9694"). InnerVolumeSpecName "kube-api-access-zrtsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:13.453451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.453420    4839 patch_prober.go:29] interesting pod/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 29 23:00:13.453805 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.453471    4839 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 29 23:00:13.498907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.498825    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75f5a631-acbe-4e70-8ce9-0a89740c9694-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:13.498907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:13.498856    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrtsw\" (UniqueName: \"kubernetes.io/projected/75f5a631-acbe-4e70-8ce9-0a89740c9694-kube-api-access-zrtsw\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:14.099660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.099622    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:14.129797 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.129768    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"
Oct 29 23:00:14.129797 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.129783    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf" event={"ID":"75f5a631-acbe-4e70-8ce9-0a89740c9694","Type":"ContainerDied","Data":"38e858b9456e8d4410ff996a172fdbde3d2520f64ac6dc09429e48bc4487a428"}
Oct 29 23:00:14.130020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.129846    4839 scope.go:122] "RemoveContainer" containerID="ec5022ab529535736db0b4807d40b218baaabf9049fa13077d723965ac0fc502"
Oct 29 23:00:14.261317 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.261232    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"]
Oct 29 23:00:14.269896 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.269842    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-34d0fd42-5b7d88f787-tnrpf"]
Oct 29 23:00:14.457745 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.457667    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 23:00:14.458110 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.458087    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 23:00:14.458867 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:14.458848    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 23:00:15.001909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:15.001852    4839 scope.go:122] "RemoveContainer" containerID="10038dc0ac45bde46e5be542c4431055e882a4eb32f256f324bc69dd1375ec65"
Oct 29 23:00:15.002241 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:15.002219    4839 scope.go:122] "RemoveContainer" containerID="dbde8f0f2dd553cce982d0d32d9eaae742bd7a9e65bd9c779ddb98fbbcbd3f6e"
Oct 29 23:00:15.010939 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:15.010916    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f5a631-acbe-4e70-8ce9-0a89740c9694" path="/var/lib/kubelet/pods/75f5a631-acbe-4e70-8ce9-0a89740c9694/volumes"
Oct 29 23:00:15.595899 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:15.579074    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:15.812180 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:15.810121    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:16.149433 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.149350    4839 generic.go:356] "Generic (PLEG): container finished" podID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerID="57f7757d7e06e0f44c45eda418d4f96e2eeeaa046394e903c7d640d90ee927af" exitCode=60
Oct 29 23:00:16.149580 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.149430    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" event={"ID":"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c","Type":"ContainerDied","Data":"57f7757d7e06e0f44c45eda418d4f96e2eeeaa046394e903c7d640d90ee927af"}
Oct 29 23:00:16.149580 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.149498    4839 scope.go:122] "RemoveContainer" containerID="10038dc0ac45bde46e5be542c4431055e882a4eb32f256f324bc69dd1375ec65"
Oct 29 23:00:16.149960 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.149937    4839 scope.go:122] "RemoveContainer" containerID="57f7757d7e06e0f44c45eda418d4f96e2eeeaa046394e903c7d640d90ee927af"
Oct 29 23:00:16.150314 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:16.150291    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn_odh-ai-gateway-infra(c69abd2b-cd07-40b8-a6d8-c4b2903bc80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c"
Oct 29 23:00:16.151708 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.151688    4839 generic.go:356] "Generic (PLEG): container finished" podID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerID="0458e2dfb490127b81ebaa5d371c281c2f39da2c0b4f1e39c3acb9c956c2b8de" exitCode=60
Oct 29 23:00:16.151802 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.151755    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" event={"ID":"b9747478-025e-41ee-84cd-ea0e11bdc24c","Type":"ContainerDied","Data":"0458e2dfb490127b81ebaa5d371c281c2f39da2c0b4f1e39c3acb9c956c2b8de"}
Oct 29 23:00:16.152351 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.152333    4839 scope.go:122] "RemoveContainer" containerID="0458e2dfb490127b81ebaa5d371c281c2f39da2c0b4f1e39c3acb9c956c2b8de"
Oct 29 23:00:16.152592 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:16.152576    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st_odh-ai-gateway-infra(b9747478-025e-41ee-84cd-ea0e11bdc24c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c"
Oct 29 23:00:16.162970 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:16.162947    4839 scope.go:122] "RemoveContainer" containerID="dbde8f0f2dd553cce982d0d32d9eaae742bd7a9e65bd9c779ddb98fbbcbd3f6e"
Oct 29 23:00:17.214641 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.214610    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"]
Oct 29 23:00:17.269484 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.269451    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"]
Oct 29 23:00:17.389810 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.389783    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:17.460729 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.460702    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-maas-api-service-ca\") pod \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") "
Oct 29 23:00:17.460909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.460769    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-kube-api-access-vq4hb\" (UniqueName: \"kubernetes.io/projected/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-kube-api-access-vq4hb\") pod \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\" (UID: \"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c\") "
Oct 29 23:00:17.461098 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.461074    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-maas-api-service-ca" pod "c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" (UID: "c69abd2b-cd07-40b8-a6d8-c4b2903bc80c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:17.461314 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.461288    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-maas-api-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:17.462932 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.462905    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-kube-api-access-vq4hb" pod "c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" (UID: "c69abd2b-cd07-40b8-a6d8-c4b2903bc80c"). InnerVolumeSpecName "kube-api-access-vq4hb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:17.483608 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.483583    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:17.562573 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.562535    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9747478-025e-41ee-84cd-ea0e11bdc24c-kube-api-access-zkg5n\" (UniqueName: \"kubernetes.io/projected/b9747478-025e-41ee-84cd-ea0e11bdc24c-kube-api-access-zkg5n\") pod \"b9747478-025e-41ee-84cd-ea0e11bdc24c\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") "
Oct 29 23:00:17.562761 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.562704    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9747478-025e-41ee-84cd-ea0e11bdc24c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9747478-025e-41ee-84cd-ea0e11bdc24c-maas-api-service-ca\") pod \"b9747478-025e-41ee-84cd-ea0e11bdc24c\" (UID: \"b9747478-025e-41ee-84cd-ea0e11bdc24c\") "
Oct 29 23:00:17.563091 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.563063    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9747478-025e-41ee-84cd-ea0e11bdc24c-maas-api-service-ca" pod "b9747478-025e-41ee-84cd-ea0e11bdc24c" (UID: "b9747478-025e-41ee-84cd-ea0e11bdc24c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:17.563453 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.563431    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq4hb\" (UniqueName: \"kubernetes.io/projected/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c-kube-api-access-vq4hb\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:17.563509 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.563461    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9747478-025e-41ee-84cd-ea0e11bdc24c-maas-api-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:17.565627 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.565594    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9747478-025e-41ee-84cd-ea0e11bdc24c-kube-api-access-zkg5n" pod "b9747478-025e-41ee-84cd-ea0e11bdc24c" (UID: "b9747478-025e-41ee-84cd-ea0e11bdc24c"). InnerVolumeSpecName "kube-api-access-zkg5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:17.581981 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.581955    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:17.672172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:17.671313    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkg5n\" (UniqueName: \"kubernetes.io/projected/b9747478-025e-41ee-84cd-ea0e11bdc24c-kube-api-access-zkg5n\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:18.002282 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.002252    4839 scope.go:122] "RemoveContainer" containerID="8b220e460dad470e2847a95599afb0d678028337a887a049188c30a9231b2379"
Oct 29 23:00:18.176324 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.176297    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"
Oct 29 23:00:18.176480 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.176338    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st" event={"ID":"b9747478-025e-41ee-84cd-ea0e11bdc24c","Type":"ContainerDied","Data":"af1ca5bf0d0efdda07367a07508c566ed2fa0d82b86fa54f2a83dd38bb42e223"}
Oct 29 23:00:18.176480 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.176405    4839 scope.go:122] "RemoveContainer" containerID="0458e2dfb490127b81ebaa5d371c281c2f39da2c0b4f1e39c3acb9c956c2b8de"
Oct 29 23:00:18.177880 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.177849    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"
Oct 29 23:00:18.177995 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.177874    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn" event={"ID":"c69abd2b-cd07-40b8-a6d8-c4b2903bc80c","Type":"ContainerDied","Data":"cf559a43028fb7fe0b819d85025a27100780d7ba7c9a523821a3d3967c2231ae"}
Oct 29 23:00:18.196918 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.196894    4839 scope.go:122] "RemoveContainer" containerID="57f7757d7e06e0f44c45eda418d4f96e2eeeaa046394e903c7d640d90ee927af"
Oct 29 23:00:18.270753 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.270725    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"]
Oct 29 23:00:18.305624 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.305591    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-28ed1cea-29888580-855st"]
Oct 29 23:00:18.372097 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.372056    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"]
Oct 29 23:00:18.389231 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.389201    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-a2e7c8-29888580-c6vfn"]
Oct 29 23:00:18.493449 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.493419    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:18.599325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599237    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"]
Oct 29 23:00:18.599826 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599811    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:18.599882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599827    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:18.599882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599840    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b1149292-e0a6-4f28-90c3-5147775bf9ba" containerName="revoke-keys"
Oct 29 23:00:18.599882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599847    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1149292-e0a6-4f28-90c3-5147775bf9ba" containerName="revoke-keys"
Oct 29 23:00:18.599882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599865    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:18.599882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599869    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:18.599882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599881    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599885    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599904    4839 cpu_manager.go:414] "containerMap: removing container" podUID="75f5a631-acbe-4e70-8ce9-0a89740c9694" containerName="payload-processing"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599909    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f5a631-acbe-4e70-8ce9-0a89740c9694" containerName="payload-processing"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599921    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.599925    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.600006    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.600014    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.600021    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1149292-e0a6-4f28-90c3-5147775bf9ba" containerName="revoke-keys"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.600028    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.600038    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:18.600166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.600043    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f5a631-acbe-4e70-8ce9-0a89740c9694" containerName="payload-processing"
Oct 29 23:00:18.603677 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.603657    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.622285 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.622175    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:18.627003 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.626971    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"]
Oct 29 23:00:18.682742 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.682709    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85cr\" (UniqueName: \"kubernetes.io/projected/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-kube-api-access-m85cr\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.682910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.682830    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.682910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.682872    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.784097 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.784068    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cr\" (UniqueName: \"kubernetes.io/projected/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-kube-api-access-m85cr\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.784232 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.784175    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.784232 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.784210    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.784892 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.784869    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.786338 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.786316    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.806628 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.806597    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85cr\" (UniqueName: \"kubernetes.io/projected/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-kube-api-access-m85cr\") pod \"maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.918435 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.918352    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:18.952908 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:18.952874    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:19.012877 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.012833    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" path="/var/lib/kubelet/pods/b9747478-025e-41ee-84cd-ea0e11bdc24c/volumes"
Oct 29 23:00:19.013451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.013429    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" path="/var/lib/kubelet/pods/c69abd2b-cd07-40b8-a6d8-c4b2903bc80c/volumes"
Oct 29 23:00:19.189653 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.189573    4839 generic.go:356] "Generic (PLEG): container finished" podID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerID="09aad41831ae8f27f8968fcda6afed0975b89acb289d89d504beaab120d16b33" exitCode=60
Oct 29 23:00:19.189811 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.189650    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" event={"ID":"9befbf1b-c732-4d0e-9a4f-2823c644240f","Type":"ContainerDied","Data":"09aad41831ae8f27f8968fcda6afed0975b89acb289d89d504beaab120d16b33"}
Oct 29 23:00:19.189811 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.189732    4839 scope.go:122] "RemoveContainer" containerID="8b220e460dad470e2847a95599afb0d678028337a887a049188c30a9231b2379"
Oct 29 23:00:19.190287 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.190263    4839 scope.go:122] "RemoveContainer" containerID="09aad41831ae8f27f8968fcda6afed0975b89acb289d89d504beaab120d16b33"
Oct 29 23:00:19.190599 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:19.190578    4839 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r_odh-ai-gateway-infra(9befbf1b-c732-4d0e-9a4f-2823c644240f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f"
Oct 29 23:00:19.238583 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:19.238557    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eb6687c_e86c_4f0d_a0b7_9e530ff74803.slice/crio-4b22538ae17fcf48b930e4c8a173a9cd1f11ffc1bbafe4d0066e006cc4ba80fe WatchSource:0}: Error finding container 4b22538ae17fcf48b930e4c8a173a9cd1f11ffc1bbafe4d0066e006cc4ba80fe: Status 404 returned error can't find the container with id 4b22538ae17fcf48b930e4c8a173a9cd1f11ffc1bbafe4d0066e006cc4ba80fe
Oct 29 23:00:19.242875 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.242850    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"]
Oct 29 23:00:19.728730 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:19.728689    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:20.202671 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.202639    4839 generic.go:356] "Generic (PLEG): container finished" podID="0eb6687c-e86c-4f0d-a0b7-9e530ff74803" containerID="84c258bfcb5ec2673963061a96e711e9144ef6c597f29c2d3e8b0d315e7ed130" exitCode=0
Oct 29 23:00:20.202804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.202726    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg" event={"ID":"0eb6687c-e86c-4f0d-a0b7-9e530ff74803","Type":"ContainerDied","Data":"84c258bfcb5ec2673963061a96e711e9144ef6c597f29c2d3e8b0d315e7ed130"}
Oct 29 23:00:20.202804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.202791    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg" event={"ID":"0eb6687c-e86c-4f0d-a0b7-9e530ff74803","Type":"ContainerStarted","Data":"4b22538ae17fcf48b930e4c8a173a9cd1f11ffc1bbafe4d0066e006cc4ba80fe"}
Oct 29 23:00:20.264561 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.264523    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"]
Oct 29 23:00:20.417930 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.417899    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:20.515631 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.515538    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9befbf1b-c732-4d0e-9a4f-2823c644240f-kube-api-access-j75kh\" (UniqueName: \"kubernetes.io/projected/9befbf1b-c732-4d0e-9a4f-2823c644240f-kube-api-access-j75kh\") pod \"9befbf1b-c732-4d0e-9a4f-2823c644240f\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") "
Oct 29 23:00:20.515900 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.515883    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9befbf1b-c732-4d0e-9a4f-2823c644240f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9befbf1b-c732-4d0e-9a4f-2823c644240f-maas-api-service-ca\") pod \"9befbf1b-c732-4d0e-9a4f-2823c644240f\" (UID: \"9befbf1b-c732-4d0e-9a4f-2823c644240f\") "
Oct 29 23:00:20.516299 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.516270    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9befbf1b-c732-4d0e-9a4f-2823c644240f-maas-api-service-ca" pod "9befbf1b-c732-4d0e-9a4f-2823c644240f" (UID: "9befbf1b-c732-4d0e-9a4f-2823c644240f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:20.516450 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.516434    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9befbf1b-c732-4d0e-9a4f-2823c644240f-maas-api-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:20.518361 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.518333    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9befbf1b-c732-4d0e-9a4f-2823c644240f-kube-api-access-j75kh" pod "9befbf1b-c732-4d0e-9a4f-2823c644240f" (UID: "9befbf1b-c732-4d0e-9a4f-2823c644240f"). InnerVolumeSpecName "kube-api-access-j75kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:20.617915 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:20.617875    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j75kh\" (UniqueName: \"kubernetes.io/projected/9befbf1b-c732-4d0e-9a4f-2823c644240f-kube-api-access-j75kh\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.215867 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.215827    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"
Oct 29 23:00:21.215867 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.215861    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r" event={"ID":"9befbf1b-c732-4d0e-9a4f-2823c644240f","Type":"ContainerDied","Data":"e70c01b0b74bf27e45c053559e500bdaadf9bcfadd42ab8fad5de78829a0b5c7"}
Oct 29 23:00:21.216353 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.215928    4839 scope.go:122] "RemoveContainer" containerID="09aad41831ae8f27f8968fcda6afed0975b89acb289d89d504beaab120d16b33"
Oct 29 23:00:21.282641 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.282604    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"]
Oct 29 23:00:21.296842 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.296810    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-34d0fd42-29888580-qz95r"]
Oct 29 23:00:21.395523 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.395496    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:21.403257 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.403230    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:21.549593 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.549501    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-kube-api-access-m85cr\" (UniqueName: \"kubernetes.io/projected/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-kube-api-access-m85cr\") pod \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") "
Oct 29 23:00:21.549731 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.549614    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-openshift-ca\") pod \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") "
Oct 29 23:00:21.549856 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.549836    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-xks-ca\") pod \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\" (UID: \"0eb6687c-e86c-4f0d-a0b7-9e530ff74803\") "
Oct 29 23:00:21.550030 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.550008    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-openshift-ca" pod "0eb6687c-e86c-4f0d-a0b7-9e530ff74803" (UID: "0eb6687c-e86c-4f0d-a0b7-9e530ff74803"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:21.550355 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.550338    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.551615 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.551593    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-kube-api-access-m85cr" pod "0eb6687c-e86c-4f0d-a0b7-9e530ff74803" (UID: "0eb6687c-e86c-4f0d-a0b7-9e530ff74803"). InnerVolumeSpecName "kube-api-access-m85cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:21.551807 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.551780    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-xks-ca" pod "0eb6687c-e86c-4f0d-a0b7-9e530ff74803" (UID: "0eb6687c-e86c-4f0d-a0b7-9e530ff74803"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:21.674173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.667314    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:21.674173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:21.667354    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m85cr\" (UniqueName: \"kubernetes.io/projected/0eb6687c-e86c-4f0d-a0b7-9e530ff74803-kube-api-access-m85cr\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:22.047617 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.035712    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:22.230654 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.230614    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg" event={"ID":"0eb6687c-e86c-4f0d-a0b7-9e530ff74803","Type":"ContainerDied","Data":"4b22538ae17fcf48b930e4c8a173a9cd1f11ffc1bbafe4d0066e006cc4ba80fe"}
Oct 29 23:00:22.230654 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.230647    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"
Oct 29 23:00:22.231087 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.230679    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b22538ae17fcf48b930e4c8a173a9cd1f11ffc1bbafe4d0066e006cc4ba80fe"
Oct 29 23:00:22.244382 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.244353    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:22.287738 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.287706    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"]
Oct 29 23:00:22.295039 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.295013    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-34d0fd42-6znqb"]
Oct 29 23:00:22.424458 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:22.412761    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:23.019328 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.019298    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39dfc716-e162-4571-ad9c-7af8f42a96b1" path="/var/lib/kubelet/pods/39dfc716-e162-4571-ad9c-7af8f42a96b1/volumes"
Oct 29 23:00:23.019701 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.019688    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" path="/var/lib/kubelet/pods/9befbf1b-c732-4d0e-9a4f-2823c644240f/volumes"
Oct 29 23:00:23.357851 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.357768    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"]
Oct 29 23:00:23.484754 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.484722    4839 scope.go:122] "RemoveContainer" containerID="06e3c6e42415c7098dcf049e2cc2998ac0ee660344e80d5ab5432174f246c215"
Oct 29 23:00:23.549123 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.549097    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:23.910362 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.910311    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:23.912660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.912636    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"]
Oct 29 23:00:23.912894 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:23.912870    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy" containerID="cri-o://861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0" gracePeriod=30
Oct 29 23:00:24.288011 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:24.287979    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:24.404177 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:24.404076    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:25.932423 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:25.932388    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:26.478776 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.478742    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:26.603557 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.603523    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:26.922024 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.921986    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"]
Oct 29 23:00:26.922602 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922588    4839 cpu_manager.go:414] "containerMap: removing container" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:26.922644 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922604    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:26.922644 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922629    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:26.922644 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922634    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:26.922737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922645    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:26.922737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922650    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:26.922737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922658    4839 cpu_manager.go:414] "containerMap: removing container" podUID="0eb6687c-e86c-4f0d-a0b7-9e530ff74803" containerName="revoke-keys"
Oct 29 23:00:26.922737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922663    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eb6687c-e86c-4f0d-a0b7-9e530ff74803" containerName="revoke-keys"
Oct 29 23:00:26.922737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922673    4839 cpu_manager.go:414] "containerMap: removing container" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:26.922737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922678    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:26.922917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922757    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c69abd2b-cd07-40b8-a6d8-c4b2903bc80c" containerName="cleanup"
Oct 29 23:00:26.922917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922766    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:26.922917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922773    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9747478-025e-41ee-84cd-ea0e11bdc24c" containerName="cleanup"
Oct 29 23:00:26.922917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922782    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eb6687c-e86c-4f0d-a0b7-9e530ff74803" containerName="revoke-keys"
Oct 29 23:00:26.922917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.922788    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:26.928464 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.928442    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:26.944775 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.944732    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"]
Oct 29 23:00:26.952034 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.952001    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:26.952201 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:26.952037    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:00:27.073680 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.073639    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9dj\" (UniqueName: \"kubernetes.io/projected/21ed2523-ddce-4e49-aed8-a61d408351b3-kube-api-access-mx9dj\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.073862 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.073715    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.074046 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.074022    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.175531 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.175448    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.175700 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.175534    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9dj\" (UniqueName: \"kubernetes.io/projected/21ed2523-ddce-4e49-aed8-a61d408351b3-kube-api-access-mx9dj\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.175700 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.175576    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.176248 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.176226    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.177844 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.177825    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.199442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.199415    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9dj\" (UniqueName: \"kubernetes.io/projected/21ed2523-ddce-4e49-aed8-a61d408351b3-kube-api-access-mx9dj\") pod \"maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.246683 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.246650    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:27.556803 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:27.556772    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ed2523_ddce_4e49_aed8_a61d408351b3.slice/crio-c6819208d78bab1939dc03c411ca2545dc37fe1eb648b74177db67bd48523cc5 WatchSource:0}: Error finding container c6819208d78bab1939dc03c411ca2545dc37fe1eb648b74177db67bd48523cc5: Status 404 returned error can't find the container with id c6819208d78bab1939dc03c411ca2545dc37fe1eb648b74177db67bd48523cc5
Oct 29 23:00:27.561360 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.561336    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"]
Oct 29 23:00:27.572555 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.572529    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:27.677913 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:27.677848    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:28.109065 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:28.109024    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:28.299573 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:28.299530    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf" event={"ID":"21ed2523-ddce-4e49-aed8-a61d408351b3","Type":"ContainerStarted","Data":"c97bb04ac83e47fc553eaf0eb92ef2302735b89cc0801ce31ec099c02caf7720"}
Oct 29 23:00:28.299770 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:28.299595    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf" event={"ID":"21ed2523-ddce-4e49-aed8-a61d408351b3","Type":"ContainerStarted","Data":"c6819208d78bab1939dc03c411ca2545dc37fe1eb648b74177db67bd48523cc5"}
Oct 29 23:00:28.360190 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:28.360123    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf" podStartSLOduration=2.360109566 podStartE2EDuration="2.360109566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:28.343216206 +0000 UTC m=+1085.446292472" watchObservedRunningTime="2026-10-29 23:00:28.360109566 +0000 UTC m=+1085.463185836"
Oct 29 23:00:29.024998 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.024965    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"]
Oct 29 23:00:29.120334 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.120285    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:29.218999 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.218971    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 23:00:29.220055 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.220035    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.307650    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-data\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.307717    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-kube-api-access-g4xbt\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-kube-api-access-g4xbt\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.307768    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-socket\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.307800    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-certs\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.307834    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-envoy\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.307914    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-istio-token\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-istio-token\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.307944    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-credential-socket\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.308010    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/596dcc57-b3bc-4de3-a806-d095be93c235-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596dcc57-b3bc-4de3-a806-d095be93c235-istio-podinfo\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.308078    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596dcc57-b3bc-4de3-a806-d095be93c235-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596dcc57-b3bc-4de3-a806-d095be93c235-istiod-ca-cert\") pod \"596dcc57-b3bc-4de3-a806-d095be93c235\" (UID: \"596dcc57-b3bc-4de3-a806-d095be93c235\") "
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.310459    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-certs" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:29.311172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.310668    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-data" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:29.317181 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.311851    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-socket" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:29.317181 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.312909    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-credential-socket" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:29.337361 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.330369    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-kube-api-access-g4xbt" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "kube-api-access-g4xbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:29.337361 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.331407    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596dcc57-b3bc-4de3-a806-d095be93c235-istiod-ca-cert" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:29.352198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.340297    4839 generic.go:356] "Generic (PLEG): container finished" podID="596dcc57-b3bc-4de3-a806-d095be93c235" containerID="861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0" exitCode=0
Oct 29 23:00:29.352198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.340520    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" event={"ID":"596dcc57-b3bc-4de3-a806-d095be93c235","Type":"ContainerDied","Data":"861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0"}
Oct 29 23:00:29.352198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.340576    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb" event={"ID":"596dcc57-b3bc-4de3-a806-d095be93c235","Type":"ContainerDied","Data":"fd06bf9beaa0b86d55b5ae63d4926d08d83ea805886256af6c7fb857c1df3f97"}
Oct 29 23:00:29.352198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.340604    4839 scope.go:122] "RemoveContainer" containerID="861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0"
Oct 29 23:00:29.352198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.340774    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"
Oct 29 23:00:29.352198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.344705    4839 generic.go:356] "Generic (PLEG): container finished" podID="21ed2523-ddce-4e49-aed8-a61d408351b3" containerID="c97bb04ac83e47fc553eaf0eb92ef2302735b89cc0801ce31ec099c02caf7720" exitCode=0
Oct 29 23:00:29.352198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.344857    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf" event={"ID":"21ed2523-ddce-4e49-aed8-a61d408351b3","Type":"ContainerDied","Data":"c97bb04ac83e47fc553eaf0eb92ef2302735b89cc0801ce31ec099c02caf7720"}
Oct 29 23:00:29.363427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.360025    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-istio-token" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:29.363427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.360645    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/596dcc57-b3bc-4de3-a806-d095be93c235-istio-podinfo" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:00:29.368886 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.367301    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-envoy" pod "596dcc57-b3bc-4de3-a806-d095be93c235" (UID: "596dcc57-b3bc-4de3-a806-d095be93c235"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:29.381578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.378535    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"]
Oct 29 23:00:29.389136 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.389113    4839 scope.go:122] "RemoveContainer" containerID="703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152"
Oct 29 23:00:29.396032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.396005    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"]
Oct 29 23:00:29.402284 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.402257    4839 scope.go:122] "RemoveContainer" containerID="861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0"
Oct 29 23:00:29.402593 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:29.402571    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0\": container with ID starting with 861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0 not found: ID does not exist" containerID="861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0"
Oct 29 23:00:29.402679 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.402603    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0"} err="failed to get container status \"861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0\": rpc error: code = NotFound desc = could not find container \"861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0\": container with ID starting with 861ee9b0f971c2be0b1475706b858cdff00007516ddf573913a767c650c1b5b0 not found: ID does not exist"
Oct 29 23:00:29.402679 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.402623    4839 scope.go:122] "RemoveContainer" containerID="703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152"
Oct 29 23:00:29.403001 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:29.402952    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152\": container with ID starting with 703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152 not found: ID does not exist" containerID="703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152"
Oct 29 23:00:29.403001 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.402970    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152"} err="failed to get container status \"703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152\": rpc error: code = NotFound desc = could not find container \"703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152\": container with ID starting with 703474a0a0dd0689f8b3c8edafdf3d0cd1621b6d27b2c83645e29b11740e3152 not found: ID does not exist"
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410695    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410726    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4xbt\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-kube-api-access-g4xbt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410742    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410756    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410769    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410781    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/596dcc57-b3bc-4de3-a806-d095be93c235-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410794    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/596dcc57-b3bc-4de3-a806-d095be93c235-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410806    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/596dcc57-b3bc-4de3-a806-d095be93c235-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.410827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.410819    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/596dcc57-b3bc-4de3-a806-d095be93c235-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:29.485756 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.485677    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:29.741222 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.734432    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"]
Oct 29 23:00:29.760184 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.758285    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-34d0fd42-openshift-default-86cdc8c47f-s5nqb"]
Oct 29 23:00:29.789941 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:29.789901    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:30.513500 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.513467    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:30.644697 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.644615    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-xks-ca\") pod \"21ed2523-ddce-4e49-aed8-a61d408351b3\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") "
Oct 29 23:00:30.644835 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.644709    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-openshift-ca\") pod \"21ed2523-ddce-4e49-aed8-a61d408351b3\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") "
Oct 29 23:00:30.644890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.644873    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ed2523-ddce-4e49-aed8-a61d408351b3-kube-api-access-mx9dj\" (UniqueName: \"kubernetes.io/projected/21ed2523-ddce-4e49-aed8-a61d408351b3-kube-api-access-mx9dj\") pod \"21ed2523-ddce-4e49-aed8-a61d408351b3\" (UID: \"21ed2523-ddce-4e49-aed8-a61d408351b3\") "
Oct 29 23:00:30.645131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.645096    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-openshift-ca" pod "21ed2523-ddce-4e49-aed8-a61d408351b3" (UID: "21ed2523-ddce-4e49-aed8-a61d408351b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:30.645497 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.645472    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:30.646867 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.646841    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-xks-ca" pod "21ed2523-ddce-4e49-aed8-a61d408351b3" (UID: "21ed2523-ddce-4e49-aed8-a61d408351b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:30.646998 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.646980    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ed2523-ddce-4e49-aed8-a61d408351b3-kube-api-access-mx9dj" pod "21ed2523-ddce-4e49-aed8-a61d408351b3" (UID: "21ed2523-ddce-4e49-aed8-a61d408351b3"). InnerVolumeSpecName "kube-api-access-mx9dj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:30.747053 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.747019    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ed2523-ddce-4e49-aed8-a61d408351b3-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:30.747053 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:30.747050    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx9dj\" (UniqueName: \"kubernetes.io/projected/21ed2523-ddce-4e49-aed8-a61d408351b3-kube-api-access-mx9dj\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:31.010791 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.010713    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" path="/var/lib/kubelet/pods/596dcc57-b3bc-4de3-a806-d095be93c235/volumes"
Oct 29 23:00:31.140398 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.140369    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:31.258452 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.258415    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:31.364306 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.364270    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf" event={"ID":"21ed2523-ddce-4e49-aed8-a61d408351b3","Type":"ContainerDied","Data":"c6819208d78bab1939dc03c411ca2545dc37fe1eb648b74177db67bd48523cc5"}
Oct 29 23:00:31.364477 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.364313    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"
Oct 29 23:00:31.364477 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.364327    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6819208d78bab1939dc03c411ca2545dc37fe1eb648b74177db67bd48523cc5"
Oct 29 23:00:31.447238 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.447168    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"]
Oct 29 23:00:31.452121 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.452088    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:31.482555 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.482524    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"]
Oct 29 23:00:31.577086 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.577050    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"]
Oct 29 23:00:31.633259 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.633171    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:31.786201 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:31.785462    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:33.239509 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:33.239475    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:00:33.329428 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:33.329395    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:34.025548 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:34.025489    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" podUID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" containerName="payload-pre-processing" containerID="cri-o://53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0" gracePeriod=25
Oct 29 23:00:35.031818 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:35.031784    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"]
Oct 29 23:00:35.067218 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:35.067178    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"]
Oct 29 23:00:35.178513 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:35.178479    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:35.360005 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:35.359914    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:35.493174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:35.489104    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:35.547224 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:35.547135    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" podUID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:37.089411 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:37.089365    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"]
Oct 29 23:00:37.120177 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:37.120134    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:37.234201 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:37.234166    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:37.350250 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:37.347086    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:37.927067 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:37.927026    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"]
Oct 29 23:00:38.017708 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:38.017662    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:38.136171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:38.135853    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:38.378864 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:38.378809    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" containerID="cri-o://195384b0297c02f3b460ebfaf40f258a8cd1d7cb428bbaec06839c21e36da927" gracePeriod=45
Oct 29 23:00:38.507285 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:38.507248    4839 generic.go:356] "Generic (PLEG): container finished" podID="9b386111-f2f1-4828-b667-ddd2b874184f" containerID="195384b0297c02f3b460ebfaf40f258a8cd1d7cb428bbaec06839c21e36da927" exitCode=0
Oct 29 23:00:38.507457 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:38.507314    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" event={"ID":"9b386111-f2f1-4828-b667-ddd2b874184f","Type":"ContainerDied","Data":"195384b0297c02f3b460ebfaf40f258a8cd1d7cb428bbaec06839c21e36da927"}
Oct 29 23:00:38.521668 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:38.521337    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused"
Oct 29 23:00:38.975366 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:38.975336    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 23:00:39.053565 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.053530    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b386111-f2f1-4828-b667-ddd2b874184f-kube-api-access-cjvx6\" (UniqueName: \"kubernetes.io/projected/9b386111-f2f1-4828-b667-ddd2b874184f-kube-api-access-cjvx6\") pod \"9b386111-f2f1-4828-b667-ddd2b874184f\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") "
Oct 29 23:00:39.053730 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.053630    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b386111-f2f1-4828-b667-ddd2b874184f-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b386111-f2f1-4828-b667-ddd2b874184f-tls-certs\") pod \"9b386111-f2f1-4828-b667-ddd2b874184f\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") "
Oct 29 23:00:39.053730 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.053686    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-dshm\") pod \"9b386111-f2f1-4828-b667-ddd2b874184f\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") "
Oct 29 23:00:39.053856 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.053774    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-home\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-home\") pod \"9b386111-f2f1-4828-b667-ddd2b874184f\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") "
Oct 29 23:00:39.053856 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.053841    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-model-cache\") pod \"9b386111-f2f1-4828-b667-ddd2b874184f\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") "
Oct 29 23:00:39.054015 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.053997    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-tmp-dir\") pod \"9b386111-f2f1-4828-b667-ddd2b874184f\" (UID: \"9b386111-f2f1-4828-b667-ddd2b874184f\") "
Oct 29 23:00:39.054080 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.054032    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-home" pod "9b386111-f2f1-4828-b667-ddd2b874184f" (UID: "9b386111-f2f1-4828-b667-ddd2b874184f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:39.054314 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.054274    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-model-cache" pod "9b386111-f2f1-4828-b667-ddd2b874184f" (UID: "9b386111-f2f1-4828-b667-ddd2b874184f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:39.054837 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.054519    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.054837 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.054547    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.055073 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.055040    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-tmp-dir" pod "9b386111-f2f1-4828-b667-ddd2b874184f" (UID: "9b386111-f2f1-4828-b667-ddd2b874184f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:39.056211 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.056187    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b386111-f2f1-4828-b667-ddd2b874184f-kube-api-access-cjvx6" pod "9b386111-f2f1-4828-b667-ddd2b874184f" (UID: "9b386111-f2f1-4828-b667-ddd2b874184f"). InnerVolumeSpecName "kube-api-access-cjvx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:39.057358 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.057334    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b386111-f2f1-4828-b667-ddd2b874184f-tls-certs" pod "9b386111-f2f1-4828-b667-ddd2b874184f" (UID: "9b386111-f2f1-4828-b667-ddd2b874184f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:39.057358 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.057343    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-dshm" pod "9b386111-f2f1-4828-b667-ddd2b874184f" (UID: "9b386111-f2f1-4828-b667-ddd2b874184f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:39.111474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.111444    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:39.188221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.181561    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvx6\" (UniqueName: \"kubernetes.io/projected/9b386111-f2f1-4828-b667-ddd2b874184f-kube-api-access-cjvx6\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.188221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.181602    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b386111-f2f1-4828-b667-ddd2b874184f-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.188221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.181619    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.188221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.181632    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b386111-f2f1-4828-b667-ddd2b874184f-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:39.516216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.516182    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g" event={"ID":"9b386111-f2f1-4828-b667-ddd2b874184f","Type":"ContainerDied","Data":"0ffbbae1ee54fccf1a791bf3ad95b0505b47f6c9f1306e450450127db20cc6b0"}
Oct 29 23:00:39.516386 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.516194    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"
Oct 29 23:00:39.516386 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.516245    4839 scope.go:122] "RemoveContainer" containerID="195384b0297c02f3b460ebfaf40f258a8cd1d7cb428bbaec06839c21e36da927"
Oct 29 23:00:39.584498 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.584468    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"]
Oct 29 23:00:39.596136 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.596107    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5bf55a/simulator-w5-5bf55a-kserve-59df96f7d8-w888g"]
Oct 29 23:00:39.719712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.719674    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:39.840213 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:39.840102    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:40.032786 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:40.032747    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" podUID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" containerName="payload-processing" containerID="cri-o://047c57715356baaf3fea68214c81703c1b931e141a9b1060ae6f8b4a94ff1a56" gracePeriod=25
Oct 29 23:00:40.067898 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:40.067861    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" podUID="a8454cc6-c9b0-4a21-a803-7088233ace04" containerName="payload-pre-processing" containerID="cri-o://cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e" gracePeriod=25
Oct 29 23:00:40.746287 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:40.746248    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"]
Oct 29 23:00:40.746843 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:40.746512    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb" podUID="0b123ccd-a354-4534-90ed-d1467b5dcc96" containerName="maas-api" containerID="cri-o://6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630" gracePeriod=30
Oct 29 23:00:40.801849 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:40.801820    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.148372 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.147581    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" path="/var/lib/kubelet/pods/9b386111-f2f1-4828-b667-ddd2b874184f/volumes"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.264948    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"]
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265851    4839 cpu_manager.go:414] "containerMap: removing container" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265872    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265914    4839 cpu_manager.go:414] "containerMap: removing container" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265923    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265934    4839 cpu_manager.go:414] "containerMap: removing container" podUID="21ed2523-ddce-4e49-aed8-a61d408351b3" containerName="revoke-keys"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265943    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ed2523-ddce-4e49-aed8-a61d408351b3" containerName="revoke-keys"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265975    4839 cpu_manager.go:414] "containerMap: removing container" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265983    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.265994    4839 cpu_manager.go:414] "containerMap: removing container" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.266002    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.266138    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="9befbf1b-c732-4d0e-9a4f-2823c644240f" containerName="cleanup"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.266173    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b386111-f2f1-4828-b667-ddd2b874184f" containerName="main"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.266184    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy"
Oct 29 23:00:41.270934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.266204    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ed2523-ddce-4e49-aed8-a61d408351b3" containerName="revoke-keys"
Oct 29 23:00:41.370776 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.369819    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.377549 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.375060    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 23:00:41.408947 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.408853    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.435251 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.435217    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"]
Oct 29 23:00:41.438055 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.438023    4839 cpu_manager.go:414] "containerMap: removing container" podUID="0b123ccd-a354-4534-90ed-d1467b5dcc96" containerName="maas-api"
Oct 29 23:00:41.438270 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.438253    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b123ccd-a354-4534-90ed-d1467b5dcc96" containerName="maas-api"
Oct 29 23:00:41.438581 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.438567    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="596dcc57-b3bc-4de3-a806-d095be93c235" containerName="istio-proxy"
Oct 29 23:00:41.438726 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.438714    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b123ccd-a354-4534-90ed-d1467b5dcc96" containerName="maas-api"
Oct 29 23:00:41.439684 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.439659    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:00:41.439789 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.439729    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:00:41.451827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.451777    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.458289 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.458262    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"]
Oct 29 23:00:41.464215 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.464123    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"]
Oct 29 23:00:41.478784 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.476374    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.535112 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.535079    4839 generic.go:356] "Generic (PLEG): container finished" podID="0b123ccd-a354-4534-90ed-d1467b5dcc96" containerID="6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630" exitCode=0
Oct 29 23:00:41.535310 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.535172    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb" event={"ID":"0b123ccd-a354-4534-90ed-d1467b5dcc96","Type":"ContainerDied","Data":"6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630"}
Oct 29 23:00:41.535310 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.535184    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"
Oct 29 23:00:41.535310 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.535267    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb" event={"ID":"0b123ccd-a354-4534-90ed-d1467b5dcc96","Type":"ContainerDied","Data":"855ef0fa8db5d72a021015801ccfa326f83bbe3761fa9a5283c06d382dbf9690"}
Oct 29 23:00:41.535310 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.535309    4839 scope.go:122] "RemoveContainer" containerID="6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630"
Oct 29 23:00:41.546380 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.546360    4839 scope.go:122] "RemoveContainer" containerID="6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630"
Oct 29 23:00:41.546679 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:41.546652    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630\": container with ID starting with 6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630 not found: ID does not exist" containerID="6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630"
Oct 29 23:00:41.546743 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.546681    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630"} err="failed to get container status \"6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630\": rpc error: code = NotFound desc = could not find container \"6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630\": container with ID starting with 6b52a8c1a1eec42f68ea9f28d62b7a98935364e868aa19db2367bab4a899a630 not found: ID does not exist"
Oct 29 23:00:41.581886 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.581854    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.583653 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.583626    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b123ccd-a354-4534-90ed-d1467b5dcc96-kube-api-access-7cdcn\" (UniqueName: \"kubernetes.io/projected/0b123ccd-a354-4534-90ed-d1467b5dcc96-kube-api-access-7cdcn\") pod \"0b123ccd-a354-4534-90ed-d1467b5dcc96\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") "
Oct 29 23:00:41.583780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.583685    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b123ccd-a354-4534-90ed-d1467b5dcc96-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b123ccd-a354-4534-90ed-d1467b5dcc96-openshift-service-ca\") pod \"0b123ccd-a354-4534-90ed-d1467b5dcc96\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") "
Oct 29 23:00:41.583879 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.583863    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-metrics-certs\") pod \"0b123ccd-a354-4534-90ed-d1467b5dcc96\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") "
Oct 29 23:00:41.583995 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.583979    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls\") pod \"0b123ccd-a354-4534-90ed-d1467b5dcc96\" (UID: \"0b123ccd-a354-4534-90ed-d1467b5dcc96\") "
Oct 29 23:00:41.584247 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.584220    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.584350 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.584330    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5t9\" (UniqueName: \"kubernetes.io/projected/c2fd88ed-b825-4be1-8489-a0a383eba610-kube-api-access-wk5t9\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.584520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.584479    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.584586 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.584533    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.584804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.584781    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9qw\" (UniqueName: \"kubernetes.io/projected/b67e9a89-ed82-4e56-93a7-73e5518c4d96-kube-api-access-wp9qw\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.584873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.584780    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b123ccd-a354-4534-90ed-d1467b5dcc96-openshift-service-ca" pod "0b123ccd-a354-4534-90ed-d1467b5dcc96" (UID: "0b123ccd-a354-4534-90ed-d1467b5dcc96"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:41.584997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.584975    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.585251 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.585211    4839 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b123ccd-a354-4534-90ed-d1467b5dcc96-openshift-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:41.587216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.587191    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b123ccd-a354-4534-90ed-d1467b5dcc96-kube-api-access-7cdcn" pod "0b123ccd-a354-4534-90ed-d1467b5dcc96" (UID: "0b123ccd-a354-4534-90ed-d1467b5dcc96"). InnerVolumeSpecName "kube-api-access-7cdcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:41.587574 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.587551    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-metrics-certs" pod "0b123ccd-a354-4534-90ed-d1467b5dcc96" (UID: "0b123ccd-a354-4534-90ed-d1467b5dcc96"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:41.588669 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.588620    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls" pod "0b123ccd-a354-4534-90ed-d1467b5dcc96" (UID: "0b123ccd-a354-4534-90ed-d1467b5dcc96"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:41.684871 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.684789    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:41.686342 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686313    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9qw\" (UniqueName: \"kubernetes.io/projected/b67e9a89-ed82-4e56-93a7-73e5518c4d96-kube-api-access-wp9qw\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.686479 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686368    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.686479 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686403    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.686479 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686447    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5t9\" (UniqueName: \"kubernetes.io/projected/c2fd88ed-b825-4be1-8489-a0a383eba610-kube-api-access-wk5t9\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.686666 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686534    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.686666 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686585    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.686759 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686734    4839 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-metrics-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:41.686759 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686752    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b123ccd-a354-4534-90ed-d1467b5dcc96-maas-api-tls\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:41.686871 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.686767    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cdcn\" (UniqueName: \"kubernetes.io/projected/0b123ccd-a354-4534-90ed-d1467b5dcc96-kube-api-access-7cdcn\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:41.687179 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.687125    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.687364 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.687340    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.688956 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.688937    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.688956 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.688944    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.710424 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.710395    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5t9\" (UniqueName: \"kubernetes.io/projected/c2fd88ed-b825-4be1-8489-a0a383eba610-kube-api-access-wk5t9\") pod \"maas-api-revoke-sub-f371fca0-r6rn8\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:41.710532 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.710517    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9qw\" (UniqueName: \"kubernetes.io/projected/b67e9a89-ed82-4e56-93a7-73e5518c4d96-kube-api-access-wp9qw\") pod \"maas-api-revoke-sub-596865a7-8fldr\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.779174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.777677    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:41.807181 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:41.799382    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:42.039430 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.039384    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"]
Oct 29 23:00:42.048268 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.048237    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-28ed1cea-6d446dbd5d-pbrmb"]
Oct 29 23:00:42.090604 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.090561    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" podUID="5943fc94-a232-47b8-9efb-01b669ed2787" containerName="payload-processing" containerID="cri-o://94a6a56bf5c5c49c09c671dcf701dbb80ea61b523b9dcbca4fc368b91cf9b978" gracePeriod=25
Oct 29 23:00:42.185572 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.185534    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:42.292001 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:42.291966    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fd88ed_b825_4be1_8489_a0a383eba610.slice/crio-efe05c956d0c71b7a0e2bff6e6739328ac81ea87d9c0a991e44388d115b72c0f WatchSource:0}: Error finding container efe05c956d0c71b7a0e2bff6e6739328ac81ea87d9c0a991e44388d115b72c0f: Status 404 returned error can't find the container with id efe05c956d0c71b7a0e2bff6e6739328ac81ea87d9c0a991e44388d115b72c0f
Oct 29 23:00:42.294996 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.294914    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"]
Oct 29 23:00:42.304915 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.304887    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"]
Oct 29 23:00:42.306420 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:42.306397    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67e9a89_ed82_4e56_93a7_73e5518c4d96.slice/crio-2be138562e8ec6c5e284751e03bdb3837d7c9fa1661d3034e05b0b9ef8f2dd6a WatchSource:0}: Error finding container 2be138562e8ec6c5e284751e03bdb3837d7c9fa1661d3034e05b0b9ef8f2dd6a: Status 404 returned error can't find the container with id 2be138562e8ec6c5e284751e03bdb3837d7c9fa1661d3034e05b0b9ef8f2dd6a
Oct 29 23:00:42.365260 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.365226    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"]
Oct 29 23:00:42.370779 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.370754    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.388238 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.387824    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"]
Oct 29 23:00:42.507587 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.507554    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.507764 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.507622    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.507764 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.507740    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn2fk\" (UniqueName: \"kubernetes.io/projected/34e35069-c61b-4d56-b47e-70b08f54dffe-kube-api-access-qn2fk\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.517968 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.517933    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" podUID="5943fc94-a232-47b8-9efb-01b669ed2787" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:42.547135 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.547041    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8" event={"ID":"c2fd88ed-b825-4be1-8489-a0a383eba610","Type":"ContainerStarted","Data":"efe05c956d0c71b7a0e2bff6e6739328ac81ea87d9c0a991e44388d115b72c0f"}
Oct 29 23:00:42.548469 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.548438    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr" event={"ID":"b67e9a89-ed82-4e56-93a7-73e5518c4d96","Type":"ContainerStarted","Data":"2be138562e8ec6c5e284751e03bdb3837d7c9fa1661d3034e05b0b9ef8f2dd6a"}
Oct 29 23:00:42.609137 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.609107    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.609284 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.609220    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.609284 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.609250    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn2fk\" (UniqueName: \"kubernetes.io/projected/34e35069-c61b-4d56-b47e-70b08f54dffe-kube-api-access-qn2fk\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.609895 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.609874    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.611654 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.611632    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.634392 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.634359    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" podUID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:42.634493 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.634389    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" podUID="a8454cc6-c9b0-4a21-a803-7088233ace04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:42.644786 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.644763    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn2fk\" (UniqueName: \"kubernetes.io/projected/34e35069-c61b-4d56-b47e-70b08f54dffe-kube-api-access-qn2fk\") pod \"maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.684909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.684884    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:42.928642 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.928550    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" podUID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" containerName="payload-pre-processing" containerID="cri-o://3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7" gracePeriod=25
Oct 29 23:00:42.987701 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:00:42.987668    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e35069_c61b_4d56_b47e_70b08f54dffe.slice/crio-24e7f656215be00c431cfe4da1931026e9deda231039d1741b5fb3322f4d7694 WatchSource:0}: Error finding container 24e7f656215be00c431cfe4da1931026e9deda231039d1741b5fb3322f4d7694: Status 404 returned error can't find the container with id 24e7f656215be00c431cfe4da1931026e9deda231039d1741b5fb3322f4d7694
Oct 29 23:00:42.990321 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:42.990295    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"]
Oct 29 23:00:43.010635 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.010610    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b123ccd-a354-4534-90ed-d1467b5dcc96" path="/var/lib/kubelet/pods/0b123ccd-a354-4534-90ed-d1467b5dcc96/volumes"
Oct 29 23:00:43.560447 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.560411    4839 generic.go:356] "Generic (PLEG): container finished" podID="c2fd88ed-b825-4be1-8489-a0a383eba610" containerID="1367aef13ca07651d6c82e1d84ef2476dedcb962acc0e3bb3323c533d120f082" exitCode=0
Oct 29 23:00:43.560652 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.560503    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8" event={"ID":"c2fd88ed-b825-4be1-8489-a0a383eba610","Type":"ContainerDied","Data":"1367aef13ca07651d6c82e1d84ef2476dedcb962acc0e3bb3323c533d120f082"}
Oct 29 23:00:43.562482 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.562456    4839 generic.go:356] "Generic (PLEG): container finished" podID="b67e9a89-ed82-4e56-93a7-73e5518c4d96" containerID="61894711ce7df2fd16183a81833109d343b86ec2eac765c32621d198af0b3860" exitCode=0
Oct 29 23:00:43.562649 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.562621    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr" event={"ID":"b67e9a89-ed82-4e56-93a7-73e5518c4d96","Type":"ContainerDied","Data":"61894711ce7df2fd16183a81833109d343b86ec2eac765c32621d198af0b3860"}
Oct 29 23:00:43.564481 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.564457    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz" event={"ID":"34e35069-c61b-4d56-b47e-70b08f54dffe","Type":"ContainerStarted","Data":"4be49e5a01162036fb66598818df6f4cc2cdb7630ffb926a93568f641b1fd07b"}
Oct 29 23:00:43.564606 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.564510    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz" event={"ID":"34e35069-c61b-4d56-b47e-70b08f54dffe","Type":"ContainerStarted","Data":"24e7f656215be00c431cfe4da1931026e9deda231039d1741b5fb3322f4d7694"}
Oct 29 23:00:43.584310 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.584284    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:43.693920 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.686851    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz" podStartSLOduration=1.6868282300000002 podStartE2EDuration="1.68682823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:43.657868384 +0000 UTC m=+1100.760944659" watchObservedRunningTime="2026-10-29 23:00:43.68682823 +0000 UTC m=+1100.789904504"
Oct 29 23:00:43.973970 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:43.973939    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:44.305756 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.305723    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:44.410557 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.410517    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" containerID="cri-o://7ce5ca1063c87859f34961c77b509a5ae5924be02c6983ccc38ce4176c56cf33" gracePeriod=45
Oct 29 23:00:44.419277 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.419250    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:44.423533 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.423501    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main" containerID="cri-o://a094204d35153ea83365c6b9d5f8572601f4fd313138de2a33175dfb306cdff0" gracePeriod=45
Oct 29 23:00:44.576050 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.575969    4839 generic.go:356] "Generic (PLEG): container finished" podID="34e35069-c61b-4d56-b47e-70b08f54dffe" containerID="4be49e5a01162036fb66598818df6f4cc2cdb7630ffb926a93568f641b1fd07b" exitCode=0
Oct 29 23:00:44.576275 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.576252    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz" event={"ID":"34e35069-c61b-4d56-b47e-70b08f54dffe","Type":"ContainerDied","Data":"4be49e5a01162036fb66598818df6f4cc2cdb7630ffb926a93568f641b1fd07b"}
Oct 29 23:00:44.578525 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.578499    4839 generic.go:356] "Generic (PLEG): container finished" podID="d2c81bda-33e7-482e-b302-806d09c7518c" containerID="a094204d35153ea83365c6b9d5f8572601f4fd313138de2a33175dfb306cdff0" exitCode=0
Oct 29 23:00:44.578644 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.578567    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" event={"ID":"d2c81bda-33e7-482e-b302-806d09c7518c","Type":"ContainerDied","Data":"a094204d35153ea83365c6b9d5f8572601f4fd313138de2a33175dfb306cdff0"}
Oct 29 23:00:44.584194 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.583953    4839 generic.go:356] "Generic (PLEG): container finished" podID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerID="7ce5ca1063c87859f34961c77b509a5ae5924be02c6983ccc38ce4176c56cf33" exitCode=0
Oct 29 23:00:44.584194 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.584177    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" event={"ID":"6dd94a17-c06d-45cc-aeda-31c9b20457cc","Type":"ContainerDied","Data":"7ce5ca1063c87859f34961c77b509a5ae5924be02c6983ccc38ce4176c56cf33"}
Oct 29 23:00:44.718782 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.718742    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 23:00:44.747205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.744245    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-home\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-home\") pod \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") "
Oct 29 23:00:44.747205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.744299    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tmp-dir\") pod \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") "
Oct 29 23:00:44.747205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.744383    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tls-certs\") pod \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") "
Oct 29 23:00:44.747205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.745013    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-home" pod "6dd94a17-c06d-45cc-aeda-31c9b20457cc" (UID: "6dd94a17-c06d-45cc-aeda-31c9b20457cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.747205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.745439    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tmp-dir" pod "6dd94a17-c06d-45cc-aeda-31c9b20457cc" (UID: "6dd94a17-c06d-45cc-aeda-31c9b20457cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.748051 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.748021    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tls-certs" pod "6dd94a17-c06d-45cc-aeda-31c9b20457cc" (UID: "6dd94a17-c06d-45cc-aeda-31c9b20457cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:44.752228 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.750832    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 23:00:44.843202 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.843137    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:44.847303 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847270    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-model-cache\") pod \"d2c81bda-33e7-482e-b302-806d09c7518c\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") "
Oct 29 23:00:44.847434 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847359    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-home\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-home\") pod \"d2c81bda-33e7-482e-b302-806d09c7518c\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") "
Oct 29 23:00:44.847434 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847409    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-dshm\") pod \"d2c81bda-33e7-482e-b302-806d09c7518c\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") "
Oct 29 23:00:44.847571 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847476    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd94a17-c06d-45cc-aeda-31c9b20457cc-kube-api-access-hxbjq\" (UniqueName: \"kubernetes.io/projected/6dd94a17-c06d-45cc-aeda-31c9b20457cc-kube-api-access-hxbjq\") pod \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") "
Oct 29 23:00:44.847571 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847513    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-model-cache\") pod \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") "
Oct 29 23:00:44.847571 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847546    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-dshm\") pod \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\" (UID: \"6dd94a17-c06d-45cc-aeda-31c9b20457cc\") "
Oct 29 23:00:44.847741 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847610    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-openshift-ca\") pod \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") "
Oct 29 23:00:44.847741 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847639    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2c81bda-33e7-482e-b302-806d09c7518c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c81bda-33e7-482e-b302-806d09c7518c-tls-certs\") pod \"d2c81bda-33e7-482e-b302-806d09c7518c\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") "
Oct 29 23:00:44.847741 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847669    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-xks-ca\") pod \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") "
Oct 29 23:00:44.847903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847762    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b67e9a89-ed82-4e56-93a7-73e5518c4d96-kube-api-access-wp9qw\" (UniqueName: \"kubernetes.io/projected/b67e9a89-ed82-4e56-93a7-73e5518c4d96-kube-api-access-wp9qw\") pod \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\" (UID: \"b67e9a89-ed82-4e56-93a7-73e5518c4d96\") "
Oct 29 23:00:44.847903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847780    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2c81bda-33e7-482e-b302-806d09c7518c-kube-api-access-fbp5c\" (UniqueName: \"kubernetes.io/projected/d2c81bda-33e7-482e-b302-806d09c7518c-kube-api-access-fbp5c\") pod \"d2c81bda-33e7-482e-b302-806d09c7518c\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") "
Oct 29 23:00:44.847903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.847857    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-tmp-dir\") pod \"d2c81bda-33e7-482e-b302-806d09c7518c\" (UID: \"d2c81bda-33e7-482e-b302-806d09c7518c\") "
Oct 29 23:00:44.848448 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.848420    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.848541 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.848476    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.848541 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.848491    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dd94a17-c06d-45cc-aeda-31c9b20457cc-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.849035 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.849008    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-tmp-dir" pod "d2c81bda-33e7-482e-b302-806d09c7518c" (UID: "d2c81bda-33e7-482e-b302-806d09c7518c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.851075 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.851051    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-model-cache" pod "d2c81bda-33e7-482e-b302-806d09c7518c" (UID: "d2c81bda-33e7-482e-b302-806d09c7518c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.852243 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.852219    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-model-cache" pod "6dd94a17-c06d-45cc-aeda-31c9b20457cc" (UID: "6dd94a17-c06d-45cc-aeda-31c9b20457cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.852425 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.852405    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-home" pod "d2c81bda-33e7-482e-b302-806d09c7518c" (UID: "d2c81bda-33e7-482e-b302-806d09c7518c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.852500 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.852487    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd94a17-c06d-45cc-aeda-31c9b20457cc-kube-api-access-hxbjq" pod "6dd94a17-c06d-45cc-aeda-31c9b20457cc" (UID: "6dd94a17-c06d-45cc-aeda-31c9b20457cc"). InnerVolumeSpecName "kube-api-access-hxbjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:44.852855 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.852834    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-openshift-ca" pod "b67e9a89-ed82-4e56-93a7-73e5518c4d96" (UID: "b67e9a89-ed82-4e56-93a7-73e5518c4d96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:44.854260 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.854085    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-xks-ca" pod "b67e9a89-ed82-4e56-93a7-73e5518c4d96" (UID: "b67e9a89-ed82-4e56-93a7-73e5518c4d96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:44.854260 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.854225    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-dshm" pod "6dd94a17-c06d-45cc-aeda-31c9b20457cc" (UID: "6dd94a17-c06d-45cc-aeda-31c9b20457cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.856578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.856225    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c81bda-33e7-482e-b302-806d09c7518c-kube-api-access-fbp5c" pod "d2c81bda-33e7-482e-b302-806d09c7518c" (UID: "d2c81bda-33e7-482e-b302-806d09c7518c"). InnerVolumeSpecName "kube-api-access-fbp5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:44.856578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.856417    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2c81bda-33e7-482e-b302-806d09c7518c-tls-certs" pod "d2c81bda-33e7-482e-b302-806d09c7518c" (UID: "d2c81bda-33e7-482e-b302-806d09c7518c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:44.856578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.856543    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-dshm" pod "d2c81bda-33e7-482e-b302-806d09c7518c" (UID: "d2c81bda-33e7-482e-b302-806d09c7518c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:44.856578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.856541    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b67e9a89-ed82-4e56-93a7-73e5518c4d96-kube-api-access-wp9qw" pod "b67e9a89-ed82-4e56-93a7-73e5518c4d96" (UID: "b67e9a89-ed82-4e56-93a7-73e5518c4d96"). InnerVolumeSpecName "kube-api-access-wp9qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:44.859897 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.859878    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:44.949072 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949041    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fd88ed-b825-4be1-8489-a0a383eba610-kube-api-access-wk5t9\" (UniqueName: \"kubernetes.io/projected/c2fd88ed-b825-4be1-8489-a0a383eba610-kube-api-access-wk5t9\") pod \"c2fd88ed-b825-4be1-8489-a0a383eba610\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") "
Oct 29 23:00:44.949282 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949081    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-openshift-ca\") pod \"c2fd88ed-b825-4be1-8489-a0a383eba610\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") "
Oct 29 23:00:44.949282 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949168    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-xks-ca\") pod \"c2fd88ed-b825-4be1-8489-a0a383eba610\" (UID: \"c2fd88ed-b825-4be1-8489-a0a383eba610\") "
Oct 29 23:00:44.949513 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949489    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-openshift-ca" pod "c2fd88ed-b825-4be1-8489-a0a383eba610" (UID: "c2fd88ed-b825-4be1-8489-a0a383eba610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:44.949828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949808    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949829    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949840    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949849    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949857    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxbjq\" (UniqueName: \"kubernetes.io/projected/6dd94a17-c06d-45cc-aeda-31c9b20457cc-kube-api-access-hxbjq\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949866    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949874    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dd94a17-c06d-45cc-aeda-31c9b20457cc-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949884    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949893    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c81bda-33e7-482e-b302-806d09c7518c-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949901    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b67e9a89-ed82-4e56-93a7-73e5518c4d96-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949912    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9qw\" (UniqueName: \"kubernetes.io/projected/b67e9a89-ed82-4e56-93a7-73e5518c4d96-kube-api-access-wp9qw\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949924    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbp5c\" (UniqueName: \"kubernetes.io/projected/d2c81bda-33e7-482e-b302-806d09c7518c-kube-api-access-fbp5c\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.949924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.949933    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c81bda-33e7-482e-b302-806d09c7518c-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:44.951388 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.951361    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-xks-ca" pod "c2fd88ed-b825-4be1-8489-a0a383eba610" (UID: "c2fd88ed-b825-4be1-8489-a0a383eba610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:44.951634 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:44.951604    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fd88ed-b825-4be1-8489-a0a383eba610-kube-api-access-wk5t9" pod "c2fd88ed-b825-4be1-8489-a0a383eba610" (UID: "c2fd88ed-b825-4be1-8489-a0a383eba610"). InnerVolumeSpecName "kube-api-access-wk5t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:45.051454 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.051428    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk5t9\" (UniqueName: \"kubernetes.io/projected/c2fd88ed-b825-4be1-8489-a0a383eba610-kube-api-access-wk5t9\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.051454 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.051452    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fd88ed-b825-4be1-8489-a0a383eba610-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.125347 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.125266    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"]
Oct 29 23:00:45.126295 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126273    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main"
Oct 29 23:00:45.126427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126414    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main"
Oct 29 23:00:45.126541 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126530    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c2fd88ed-b825-4be1-8489-a0a383eba610" containerName="revoke-keys"
Oct 29 23:00:45.126609 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126600    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fd88ed-b825-4be1-8489-a0a383eba610" containerName="revoke-keys"
Oct 29 23:00:45.126700 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126691    4839 cpu_manager.go:414] "containerMap: removing container" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main"
Oct 29 23:00:45.126764 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126756    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main"
Oct 29 23:00:45.126829 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126821    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b67e9a89-ed82-4e56-93a7-73e5518c4d96" containerName="revoke-keys"
Oct 29 23:00:45.126897 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.126889    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b67e9a89-ed82-4e56-93a7-73e5518c4d96" containerName="revoke-keys"
Oct 29 23:00:45.127085 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.127073    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" containerName="main"
Oct 29 23:00:45.127179 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.127169    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fd88ed-b825-4be1-8489-a0a383eba610" containerName="revoke-keys"
Oct 29 23:00:45.127264 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.127255    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main"
Oct 29 23:00:45.127338 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.127329    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b67e9a89-ed82-4e56-93a7-73e5518c4d96" containerName="revoke-keys"
Oct 29 23:00:45.153412 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.153377    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.155534 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.155500    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"]
Oct 29 23:00:45.254698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.254662    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.254848 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.254796    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.254913 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.254852    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h656\" (UniqueName: \"kubernetes.io/projected/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-kube-api-access-4h656\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.355831 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.355797    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.356028 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.355919    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.356028 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.355961    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h656\" (UniqueName: \"kubernetes.io/projected/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-kube-api-access-4h656\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.356498 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.356470    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.358333 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.358306    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.377425 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.377369    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h656\" (UniqueName: \"kubernetes.io/projected/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-kube-api-access-4h656\") pod \"maas-api-revoke-sub-98d8025f-s8vq6\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.467589 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.467550    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:45.536557 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.536516    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" podUID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:45.595889 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.595855    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"
Oct 29 23:00:45.596105 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.595854    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8" event={"ID":"c2fd88ed-b825-4be1-8489-a0a383eba610","Type":"ContainerDied","Data":"efe05c956d0c71b7a0e2bff6e6739328ac81ea87d9c0a991e44388d115b72c0f"}
Oct 29 23:00:45.596105 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.595994    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efe05c956d0c71b7a0e2bff6e6739328ac81ea87d9c0a991e44388d115b72c0f"
Oct 29 23:00:45.597756 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.597732    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"
Oct 29 23:00:45.597907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.597770    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr" event={"ID":"b67e9a89-ed82-4e56-93a7-73e5518c4d96","Type":"ContainerDied","Data":"2be138562e8ec6c5e284751e03bdb3837d7c9fa1661d3034e05b0b9ef8f2dd6a"}
Oct 29 23:00:45.597907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.597843    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be138562e8ec6c5e284751e03bdb3837d7c9fa1661d3034e05b0b9ef8f2dd6a"
Oct 29 23:00:45.599917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.599894    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx" event={"ID":"d2c81bda-33e7-482e-b302-806d09c7518c","Type":"ContainerDied","Data":"3ea9ae24c4be2025326d451375ec2cd6e67f2366d1f5c61eddac51a79c480b76"}
Oct 29 23:00:45.600094 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.599950    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"
Oct 29 23:00:45.600254 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.600084    4839 scope.go:122] "RemoveContainer" containerID="a094204d35153ea83365c6b9d5f8572601f4fd313138de2a33175dfb306cdff0"
Oct 29 23:00:45.614442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.614412    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"
Oct 29 23:00:45.614605 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.614469    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" event={"ID":"6dd94a17-c06d-45cc-aeda-31c9b20457cc","Type":"ContainerDied","Data":"b0951d3700c96a8d698478757cb953729939c1908f45a8ad64349d9af9aae63d"}
Oct 29 23:00:45.617081 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.617051    4839 scope.go:122] "RemoveContainer" containerID="7ce5ca1063c87859f34961c77b509a5ae5924be02c6983ccc38ce4176c56cf33"
Oct 29 23:00:45.670761 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.670676    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"]
Oct 29 23:00:45.680997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.680849    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:00:45.685892 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.685859    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/simulator-w2-a2e7c8-kserve-68d99944f-5d7dx"]
Oct 29 23:00:45.721439 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.721404    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:45.761325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.760454    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:45.768238 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.761922    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-openshift-ca\") pod \"34e35069-c61b-4d56-b47e-70b08f54dffe\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") "
Oct 29 23:00:45.768238 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.762069    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e35069-c61b-4d56-b47e-70b08f54dffe-kube-api-access-qn2fk\" (UniqueName: \"kubernetes.io/projected/34e35069-c61b-4d56-b47e-70b08f54dffe-kube-api-access-qn2fk\") pod \"34e35069-c61b-4d56-b47e-70b08f54dffe\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") "
Oct 29 23:00:45.768238 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.762253    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-xks-ca\") pod \"34e35069-c61b-4d56-b47e-70b08f54dffe\" (UID: \"34e35069-c61b-4d56-b47e-70b08f54dffe\") "
Oct 29 23:00:45.768238 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.765896    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-openshift-ca" pod "34e35069-c61b-4d56-b47e-70b08f54dffe" (UID: "34e35069-c61b-4d56-b47e-70b08f54dffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:45.771195 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.771104    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e35069-c61b-4d56-b47e-70b08f54dffe-kube-api-access-qn2fk" pod "34e35069-c61b-4d56-b47e-70b08f54dffe" (UID: "34e35069-c61b-4d56-b47e-70b08f54dffe"). InnerVolumeSpecName "kube-api-access-qn2fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:45.776921 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.775996    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-xks-ca" pod "34e35069-c61b-4d56-b47e-70b08f54dffe" (UID: "34e35069-c61b-4d56-b47e-70b08f54dffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:45.818276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.818226    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"]
Oct 29 23:00:45.823776 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.823712    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-a2e7c8/distinct-w2-a2e7c8-kserve-66d56fff79-77n7m"]
Oct 29 23:00:45.831519 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.831481    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"]
Oct 29 23:00:45.873166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.868401    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.873166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.868437    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn2fk\" (UniqueName: \"kubernetes.io/projected/34e35069-c61b-4d56-b47e-70b08f54dffe-kube-api-access-qn2fk\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:45.873166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:45.868454    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34e35069-c61b-4d56-b47e-70b08f54dffe-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.475447 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.475349    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main" containerID="cri-o://aa96d0207fde70f2936a0f082c9c02fe7b55c57a664c2f97749782d982d4c12d" gracePeriod=45
Oct 29 23:00:46.502777 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.502715    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" containerID="cri-o://6154ad4f401294c8e5c56c63b7db2a838e06d04d26ee4a2c3351c328b7410dec" gracePeriod=45
Oct 29 23:00:46.597064 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.597028    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 29 23:00:46.598330 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.598301    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main" containerID="cri-o://7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4" gracePeriod=45
Oct 29 23:00:46.624506 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.624474    4839 generic.go:356] "Generic (PLEG): container finished" podID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerID="6154ad4f401294c8e5c56c63b7db2a838e06d04d26ee4a2c3351c328b7410dec" exitCode=0
Oct 29 23:00:46.624674 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.624534    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" event={"ID":"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3","Type":"ContainerDied","Data":"6154ad4f401294c8e5c56c63b7db2a838e06d04d26ee4a2c3351c328b7410dec"}
Oct 29 23:00:46.626413 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.626386    4839 generic.go:356] "Generic (PLEG): container finished" podID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerID="aa96d0207fde70f2936a0f082c9c02fe7b55c57a664c2f97749782d982d4c12d" exitCode=0
Oct 29 23:00:46.626546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.626462    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" event={"ID":"e3f0c9cf-92b3-4b5f-9818-1643ce63672f","Type":"ContainerDied","Data":"aa96d0207fde70f2936a0f082c9c02fe7b55c57a664c2f97749782d982d4c12d"}
Oct 29 23:00:46.630135 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.630107    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz" event={"ID":"34e35069-c61b-4d56-b47e-70b08f54dffe","Type":"ContainerDied","Data":"24e7f656215be00c431cfe4da1931026e9deda231039d1741b5fb3322f4d7694"}
Oct 29 23:00:46.630135 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.630126    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"
Oct 29 23:00:46.630314 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.630201    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24e7f656215be00c431cfe4da1931026e9deda231039d1741b5fb3322f4d7694"
Oct 29 23:00:46.631964 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.631938    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6" event={"ID":"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e","Type":"ContainerStarted","Data":"602e55e23541b1dc3607f10a3f67c4f77b01825db38c74b38cd3e97cd4afcd25"}
Oct 29 23:00:46.632084 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.631993    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6" event={"ID":"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e","Type":"ContainerStarted","Data":"09ea9a3c1a4934ea8440fad18eade8ec91a4adde633c3853d448cbb3588f34bb"}
Oct 29 23:00:46.679744 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.679688    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6" podStartSLOduration=1.6796699290000001 podStartE2EDuration="1.679669929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:00:46.677711691 +0000 UTC m=+1103.780787962" watchObservedRunningTime="2026-10-29 23:00:46.679669929 +0000 UTC m=+1103.782746201"
Oct 29 23:00:46.763066 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.763039    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.779903    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tls-certs\") pod \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") "
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.780045    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tmp-dir\") pod \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") "
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.780100    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-dshm\") pod \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") "
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.780118    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-home\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-home\") pod \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") "
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.780137    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-model-cache\") pod \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") "
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.780219    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-kube-api-access-c98q4\" (UniqueName: \"kubernetes.io/projected/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-kube-api-access-c98q4\") pod \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\" (UID: \"e3f0c9cf-92b3-4b5f-9818-1643ce63672f\") "
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.783540    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-home" pod "e3f0c9cf-92b3-4b5f-9818-1643ce63672f" (UID: "e3f0c9cf-92b3-4b5f-9818-1643ce63672f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.783728    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-model-cache" pod "e3f0c9cf-92b3-4b5f-9818-1643ce63672f" (UID: "e3f0c9cf-92b3-4b5f-9818-1643ce63672f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.783903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.783867    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tmp-dir" pod "e3f0c9cf-92b3-4b5f-9818-1643ce63672f" (UID: "e3f0c9cf-92b3-4b5f-9818-1643ce63672f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.786632 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.786602    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-dshm" pod "e3f0c9cf-92b3-4b5f-9818-1643ce63672f" (UID: "e3f0c9cf-92b3-4b5f-9818-1643ce63672f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.787559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.787531    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-kube-api-access-c98q4" pod "e3f0c9cf-92b3-4b5f-9818-1643ce63672f" (UID: "e3f0c9cf-92b3-4b5f-9818-1643ce63672f"). InnerVolumeSpecName "kube-api-access-c98q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.787657 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.787576    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tls-certs" pod "e3f0c9cf-92b3-4b5f-9818-1643ce63672f" (UID: "e3f0c9cf-92b3-4b5f-9818-1643ce63672f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:46.884559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.882555    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98q4\" (UniqueName: \"kubernetes.io/projected/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-kube-api-access-c98q4\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.884559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.882586    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.884559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.882599    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.884559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.882610    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.884559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.882622    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.884559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.882635    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3f0c9cf-92b3-4b5f-9818-1643ce63672f-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.911060 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.911035    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 23:00:46.913195 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.913137    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 23:00:46.983958 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.983921    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tls-certs\") pod \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") "
Oct 29 23:00:46.983958 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.983973    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5397ab93-f411-494f-a840-5bfea3e38c39-tls-certs\" (UniqueName: \"kubernetes.io/secret/5397ab93-f411-494f-a840-5bfea3e38c39-tls-certs\") pod \"5397ab93-f411-494f-a840-5bfea3e38c39\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") "
Oct 29 23:00:46.984241 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984029    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-dshm\") pod \"5397ab93-f411-494f-a840-5bfea3e38c39\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") "
Oct 29 23:00:46.984241 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984118    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-dshm\") pod \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") "
Oct 29 23:00:46.984241 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984217    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-home\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-home\") pod \"5397ab93-f411-494f-a840-5bfea3e38c39\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") "
Oct 29 23:00:46.984395 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984285    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tmp-dir\") pod \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") "
Oct 29 23:00:46.984395 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984358    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5397ab93-f411-494f-a840-5bfea3e38c39-kube-api-access-jhvrt\" (UniqueName: \"kubernetes.io/projected/5397ab93-f411-494f-a840-5bfea3e38c39-kube-api-access-jhvrt\") pod \"5397ab93-f411-494f-a840-5bfea3e38c39\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") "
Oct 29 23:00:46.984520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984404    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-model-cache\") pod \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") "
Oct 29 23:00:46.984520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984437    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-tmp-dir\") pod \"5397ab93-f411-494f-a840-5bfea3e38c39\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") "
Oct 29 23:00:46.984520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984475    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-home\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-home\") pod \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") "
Oct 29 23:00:46.984520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984507    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-kube-api-access-hhzj7\" (UniqueName: \"kubernetes.io/projected/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-kube-api-access-hhzj7\") pod \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\" (UID: \"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3\") "
Oct 29 23:00:46.984718 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984555    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-model-cache\") pod \"5397ab93-f411-494f-a840-5bfea3e38c39\" (UID: \"5397ab93-f411-494f-a840-5bfea3e38c39\") "
Oct 29 23:00:46.984718 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984605    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-home" pod "5397ab93-f411-494f-a840-5bfea3e38c39" (UID: "5397ab93-f411-494f-a840-5bfea3e38c39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.984815 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984799    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tmp-dir" pod "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" (UID: "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.985009 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.984978    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-model-cache" pod "5397ab93-f411-494f-a840-5bfea3e38c39" (UID: "5397ab93-f411-494f-a840-5bfea3e38c39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.985502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985160    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-tmp-dir" pod "5397ab93-f411-494f-a840-5bfea3e38c39" (UID: "5397ab93-f411-494f-a840-5bfea3e38c39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.985502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985169    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-home" pod "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" (UID: "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.985502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985470    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-model-cache" pod "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" (UID: "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.985723 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985598    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.985723 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985642    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.985723 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985658    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.985723 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985670    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.985723 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.985682    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:46.990422 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.990390    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5397ab93-f411-494f-a840-5bfea3e38c39-tls-certs" pod "5397ab93-f411-494f-a840-5bfea3e38c39" (UID: "5397ab93-f411-494f-a840-5bfea3e38c39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:46.990550 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.990457    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-dshm" pod "5397ab93-f411-494f-a840-5bfea3e38c39" (UID: "5397ab93-f411-494f-a840-5bfea3e38c39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.990610 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.990570    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5397ab93-f411-494f-a840-5bfea3e38c39-kube-api-access-jhvrt" pod "5397ab93-f411-494f-a840-5bfea3e38c39" (UID: "5397ab93-f411-494f-a840-5bfea3e38c39"). InnerVolumeSpecName "kube-api-access-jhvrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.990672 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.990603    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-kube-api-access-hhzj7" pod "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" (UID: "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3"). InnerVolumeSpecName "kube-api-access-hhzj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:46.990910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.990893    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-dshm" pod "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" (UID: "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:00:46.991229 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:46.991129    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tls-certs" pod "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" (UID: "bfda4363-0693-4a7e-8ea5-8babe3a0bfe3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:47.012032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.011996    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd94a17-c06d-45cc-aeda-31c9b20457cc" path="/var/lib/kubelet/pods/6dd94a17-c06d-45cc-aeda-31c9b20457cc/volumes"
Oct 29 23:00:47.015483 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.012661    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2c81bda-33e7-482e-b302-806d09c7518c" path="/var/lib/kubelet/pods/d2c81bda-33e7-482e-b302-806d09c7518c/volumes"
Oct 29 23:00:47.087131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.086848    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5397ab93-f411-494f-a840-5bfea3e38c39-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.087131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.086873    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5397ab93-f411-494f-a840-5bfea3e38c39-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.087131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.086882    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.087131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.086892    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhvrt\" (UniqueName: \"kubernetes.io/projected/5397ab93-f411-494f-a840-5bfea3e38c39-kube-api-access-jhvrt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.087131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.086902    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.087131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.086912    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzj7\" (UniqueName: \"kubernetes.io/projected/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-kube-api-access-hhzj7\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.087131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.086921    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:47.223652 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.223620    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"]
Oct 29 23:00:47.413183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.413123    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:47.614172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.604643    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:47.680708 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.680633    4839 generic.go:356] "Generic (PLEG): container finished" podID="5397ab93-f411-494f-a840-5bfea3e38c39" containerID="7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4" exitCode=0
Oct 29 23:00:47.681243 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.681038    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"
Oct 29 23:00:47.688210 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.681056    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" event={"ID":"5397ab93-f411-494f-a840-5bfea3e38c39","Type":"ContainerDied","Data":"7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4"}
Oct 29 23:00:47.688210 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.686916    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm" event={"ID":"5397ab93-f411-494f-a840-5bfea3e38c39","Type":"ContainerDied","Data":"0af5a2526cd75437a073e62c6b63ece868b4e9522023c8c3d68745186684cc1b"}
Oct 29 23:00:47.688210 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.686950    4839 scope.go:122] "RemoveContainer" containerID="7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4"
Oct 29 23:00:47.715571 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.715527    4839 generic.go:356] "Generic (PLEG): container finished" podID="a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" containerID="602e55e23541b1dc3607f10a3f67c4f77b01825db38c74b38cd3e97cd4afcd25" exitCode=0
Oct 29 23:00:47.715743 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.715719    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6" event={"ID":"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e","Type":"ContainerDied","Data":"602e55e23541b1dc3607f10a3f67c4f77b01825db38c74b38cd3e97cd4afcd25"}
Oct 29 23:00:47.728826 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.723916    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r" event={"ID":"bfda4363-0693-4a7e-8ea5-8babe3a0bfe3","Type":"ContainerDied","Data":"86de31eb852a0bccb4a5a00ed868348d077d112d989c7383b2009da811d61668"}
Oct 29 23:00:47.728826 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.724220    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"
Oct 29 23:00:47.747488 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.733387    4839 scope.go:122] "RemoveContainer" containerID="7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4"
Oct 29 23:00:47.747488 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:47.734086    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4\": container with ID starting with 7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4 not found: ID does not exist" containerID="7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4"
Oct 29 23:00:47.747488 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.734138    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4"} err="failed to get container status \"7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4\": rpc error: code = NotFound desc = could not find container \"7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4\": container with ID starting with 7c8271ab68b29f4f679ee48e8cf01d89e56debfbbe39ceb116bc6be8882525c4 not found: ID does not exist"
Oct 29 23:00:47.747488 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.734259    4839 scope.go:122] "RemoveContainer" containerID="6154ad4f401294c8e5c56c63b7db2a838e06d04d26ee4a2c3351c328b7410dec"
Oct 29 23:00:47.747488 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.739330    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2" event={"ID":"e3f0c9cf-92b3-4b5f-9818-1643ce63672f","Type":"ContainerDied","Data":"a666cee1778e2495f51f9ac4d54d0c069396b998a55a7fe1c5080229c2168274"}
Oct 29 23:00:47.747488 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.739429    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"
Oct 29 23:00:47.766999 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:47.766635    4839 scope.go:122] "RemoveContainer" containerID="aa96d0207fde70f2936a0f082c9c02fe7b55c57a664c2f97749782d982d4c12d"
Oct 29 23:00:48.004298 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.004215    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:48.049167 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.046927    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"]
Oct 29 23:00:48.057429 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.057399    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-20f88f/premium-w1-20f88f-kserve-7855b9b979-tq4gm"]
Oct 29 23:00:48.103344 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.103308    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"]
Oct 29 23:00:48.110408 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.110379    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/simulator-w6-81f92d-kserve-6684576b69-99t7r"]
Oct 29 23:00:48.182932 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.182861    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:48.197166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.191218    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"]
Oct 29 23:00:48.215077 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.215040    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-81f92d/distinct-2-w6-81f92d-kserve-689579d7d4-lbqf2"]
Oct 29 23:00:48.227200 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.227127    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"]
Oct 29 23:00:48.231120 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.231093    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-5bf55a-27fqg"]
Oct 29 23:00:48.386181 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.376138    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:48.714109 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.714027    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:48.913959 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.913932    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:48.926676 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.926646    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-kube-api-access-4h656\" (UniqueName: \"kubernetes.io/projected/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-kube-api-access-4h656\") pod \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") "
Oct 29 23:00:48.926804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.926734    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-openshift-ca\") pod \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") "
Oct 29 23:00:48.926909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.926892    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-xks-ca\") pod \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\" (UID: \"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e\") "
Oct 29 23:00:48.927139 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.927112    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-openshift-ca" pod "a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" (UID: "a8af9217-f7d5-4aa6-acc2-58f9aef83b6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:48.927448 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.927430    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:48.928843 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.928817    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-xks-ca" pod "a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" (UID: "a8af9217-f7d5-4aa6-acc2-58f9aef83b6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:00:48.928944 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:48.928923    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-kube-api-access-4h656" pod "a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" (UID: "a8af9217-f7d5-4aa6-acc2-58f9aef83b6e"). InnerVolumeSpecName "kube-api-access-4h656". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:49.011424 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.011340    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eb6687c-e86c-4f0d-a0b7-9e530ff74803" path="/var/lib/kubelet/pods/0eb6687c-e86c-4f0d-a0b7-9e530ff74803/volumes"
Oct 29 23:00:49.011831 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.011817    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" path="/var/lib/kubelet/pods/5397ab93-f411-494f-a840-5bfea3e38c39/volumes"
Oct 29 23:00:49.012269 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.012255    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" path="/var/lib/kubelet/pods/bfda4363-0693-4a7e-8ea5-8babe3a0bfe3/volumes"
Oct 29 23:00:49.012939 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.012922    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" path="/var/lib/kubelet/pods/e3f0c9cf-92b3-4b5f-9818-1643ce63672f/volumes"
Oct 29 23:00:49.028428 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.028408    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:49.028535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.028432    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h656\" (UniqueName: \"kubernetes.io/projected/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e-kube-api-access-4h656\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:49.759996 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.759966    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"
Oct 29 23:00:49.760425 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.760004    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6" event={"ID":"a8af9217-f7d5-4aa6-acc2-58f9aef83b6e","Type":"ContainerDied","Data":"09ea9a3c1a4934ea8440fad18eade8ec91a4adde633c3853d448cbb3588f34bb"}
Oct 29 23:00:49.760425 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:49.760052    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09ea9a3c1a4934ea8440fad18eade8ec91a4adde633c3853d448cbb3588f34bb"
Oct 29 23:00:50.189945 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:50.189907    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:50.582481 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:50.582443    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" podUID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:51.685340 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:51.685246    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:52.079568 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:52.079535    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:52.517658 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:52.517612    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" podUID="5943fc94-a232-47b8-9efb-01b669ed2787" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:52.634074 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:52.634039    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" podUID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:52.634236 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:52.634097    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" podUID="a8454cc6-c9b0-4a21-a803-7088233ace04" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:00:52.896819 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:52.896733    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:53.039660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:53.038682    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:53.561473 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:53.560749    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:54.106640 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.106565    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:54.333580 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.333464    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:54.383827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.383799    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 23:00:54.422621 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.422586    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db34ef8b-313c-4614-b15d-09a6ac3dbe63-kube-api-access-bxsgr\" (UniqueName: \"kubernetes.io/projected/db34ef8b-313c-4614-b15d-09a6ac3dbe63-kube-api-access-bxsgr\") pod \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") "
Oct 29 23:00:54.425133 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.425102    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db34ef8b-313c-4614-b15d-09a6ac3dbe63-kube-api-access-bxsgr" pod "db34ef8b-313c-4614-b15d-09a6ac3dbe63" (UID: "db34ef8b-313c-4614-b15d-09a6ac3dbe63"). InnerVolumeSpecName "kube-api-access-bxsgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:00:54.523640 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.523604    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db34ef8b-313c-4614-b15d-09a6ac3dbe63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34ef8b-313c-4614-b15d-09a6ac3dbe63-plugins-config-volume\") pod \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\" (UID: \"db34ef8b-313c-4614-b15d-09a6ac3dbe63\") "
Oct 29 23:00:54.523955 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.523930    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db34ef8b-313c-4614-b15d-09a6ac3dbe63-plugins-config-volume" pod "db34ef8b-313c-4614-b15d-09a6ac3dbe63" (UID: "db34ef8b-313c-4614-b15d-09a6ac3dbe63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:00:54.524200 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.524185    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxsgr\" (UniqueName: \"kubernetes.io/projected/db34ef8b-313c-4614-b15d-09a6ac3dbe63-kube-api-access-bxsgr\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:54.524320 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.524203    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db34ef8b-313c-4614-b15d-09a6ac3dbe63-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:00:54.670257 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.663642    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:54.866721 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.866677    4839 generic.go:356] "Generic (PLEG): container finished" podID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" containerID="53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0" exitCode=0
Oct 29 23:00:54.866916 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.866859    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"
Oct 29 23:00:54.867100 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.867080    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" event={"ID":"db34ef8b-313c-4614-b15d-09a6ac3dbe63","Type":"ContainerDied","Data":"53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0"}
Oct 29 23:00:54.867230 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.867212    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk" event={"ID":"db34ef8b-313c-4614-b15d-09a6ac3dbe63","Type":"ContainerDied","Data":"ace517bf5b3a52b44563ee959bcdea230748ba00cbfb5647ea6992df52012b37"}
Oct 29 23:00:54.867314 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.867267    4839 scope.go:122] "RemoveContainer" containerID="53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0"
Oct 29 23:00:54.882410 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.882388    4839 scope.go:122] "RemoveContainer" containerID="53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0"
Oct 29 23:00:54.882902 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:00:54.882885    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0\": container with ID starting with 53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0 not found: ID does not exist" containerID="53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0"
Oct 29 23:00:54.883035 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.883005    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0"} err="failed to get container status \"53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0\": rpc error: code = NotFound desc = could not find container \"53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0\": container with ID starting with 53998388a439dbed4f30388620cdf7c468bbc344d48c9d08dbb694000d3271c0 not found: ID does not exist"
Oct 29 23:00:54.953629 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.953541    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"]
Oct 29 23:00:54.971183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:54.971131    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5bf55a-6db7cd9775-6rknk"]
Oct 29 23:00:55.009673 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:55.009639    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" path="/var/lib/kubelet/pods/db34ef8b-313c-4614-b15d-09a6ac3dbe63/volumes"
Oct 29 23:00:55.080159 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:55.080124    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:55.259859 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:55.259778    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:55.435718 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:55.435684    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:55.731392 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:55.731351    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:57.112663 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:57.094848    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:57.464838 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:57.464721    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:57.737257 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:57.712668    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:58.155172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:58.123243    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"]
Oct 29 23:00:58.253635 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:58.253567    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" containerID="cri-o://da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f" gracePeriod=30
Oct 29 23:00:58.353727 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:58.353687    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:58.388330 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:58.388222    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"]
Oct 29 23:00:58.450230 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:58.412265    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-20f88f-wp7wf"]
Oct 29 23:00:58.718171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:58.718044    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:00:59.067191 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:00:59.066774    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ed2523-ddce-4e49-aed8-a61d408351b3" path="/var/lib/kubelet/pods/21ed2523-ddce-4e49-aed8-a61d408351b3/volumes"
Oct 29 23:01:00.239288 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.239257    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:00.244734 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.244704    4839 generic.go:356] "Generic (PLEG): container finished" podID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" containerID="047c57715356baaf3fea68214c81703c1b931e141a9b1060ae6f8b4a94ff1a56" exitCode=0
Oct 29 23:01:00.244992 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.244969    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" event={"ID":"a48caf36-726a-4a6a-8dbb-990fb2c03bff","Type":"ContainerDied","Data":"047c57715356baaf3fea68214c81703c1b931e141a9b1060ae6f8b4a94ff1a56"}
Oct 29 23:01:00.460171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.446132    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:00.613785 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.612896    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" podUID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:01:00.636925 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.636894    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 23:01:00.673506 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.673479    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 23:01:00.711853 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.711807    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8454cc6-c9b0-4a21-a803-7088233ace04-kube-api-access-kw4nh\" (UniqueName: \"kubernetes.io/projected/a8454cc6-c9b0-4a21-a803-7088233ace04-kube-api-access-kw4nh\") pod \"a8454cc6-c9b0-4a21-a803-7088233ace04\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") "
Oct 29 23:01:00.712019 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.711883    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8454cc6-c9b0-4a21-a803-7088233ace04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8454cc6-c9b0-4a21-a803-7088233ace04-plugins-config-volume\") pod \"a8454cc6-c9b0-4a21-a803-7088233ace04\" (UID: \"a8454cc6-c9b0-4a21-a803-7088233ace04\") "
Oct 29 23:01:00.712108 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.712070    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48caf36-726a-4a6a-8dbb-990fb2c03bff-kube-api-access-hx7rk\" (UniqueName: \"kubernetes.io/projected/a48caf36-726a-4a6a-8dbb-990fb2c03bff-kube-api-access-hx7rk\") pod \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") "
Oct 29 23:01:00.712223 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.712204    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48caf36-726a-4a6a-8dbb-990fb2c03bff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48caf36-726a-4a6a-8dbb-990fb2c03bff-plugins-config-volume\") pod \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\" (UID: \"a48caf36-726a-4a6a-8dbb-990fb2c03bff\") "
Oct 29 23:01:00.712362 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.712335    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8454cc6-c9b0-4a21-a803-7088233ace04-plugins-config-volume" pod "a8454cc6-c9b0-4a21-a803-7088233ace04" (UID: "a8454cc6-c9b0-4a21-a803-7088233ace04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:00.712729 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.712704    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8454cc6-c9b0-4a21-a803-7088233ace04-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.713020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.712993    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48caf36-726a-4a6a-8dbb-990fb2c03bff-plugins-config-volume" pod "a48caf36-726a-4a6a-8dbb-990fb2c03bff" (UID: "a48caf36-726a-4a6a-8dbb-990fb2c03bff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:00.714960 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.714930    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48caf36-726a-4a6a-8dbb-990fb2c03bff-kube-api-access-hx7rk" pod "a48caf36-726a-4a6a-8dbb-990fb2c03bff" (UID: "a48caf36-726a-4a6a-8dbb-990fb2c03bff"). InnerVolumeSpecName "kube-api-access-hx7rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:00.715320 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.715296    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8454cc6-c9b0-4a21-a803-7088233ace04-kube-api-access-kw4nh" pod "a8454cc6-c9b0-4a21-a803-7088233ace04" (UID: "a8454cc6-c9b0-4a21-a803-7088233ace04"). InnerVolumeSpecName "kube-api-access-kw4nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:00.813895 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.813852    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx7rk\" (UniqueName: \"kubernetes.io/projected/a48caf36-726a-4a6a-8dbb-990fb2c03bff-kube-api-access-hx7rk\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.813895 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.813891    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a48caf36-726a-4a6a-8dbb-990fb2c03bff-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:00.814094 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:00.813905    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw4nh\" (UniqueName: \"kubernetes.io/projected/a8454cc6-c9b0-4a21-a803-7088233ace04-kube-api-access-kw4nh\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:01.254022 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.253984    4839 generic.go:356] "Generic (PLEG): container finished" podID="a8454cc6-c9b0-4a21-a803-7088233ace04" containerID="cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e" exitCode=0
Oct 29 23:01:01.254492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.254041    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"
Oct 29 23:01:01.254492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.254061    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" event={"ID":"a8454cc6-c9b0-4a21-a803-7088233ace04","Type":"ContainerDied","Data":"cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e"}
Oct 29 23:01:01.254492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.254111    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5" event={"ID":"a8454cc6-c9b0-4a21-a803-7088233ace04","Type":"ContainerDied","Data":"ebebc9dced772a42a9c9a4bc8168e9368455f134ee8e0c396764678516e58ac3"}
Oct 29 23:01:01.254492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.254139    4839 scope.go:122] "RemoveContainer" containerID="cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e"
Oct 29 23:01:01.255817 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.255793    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"
Oct 29 23:01:01.255925 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.255839    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v" event={"ID":"a48caf36-726a-4a6a-8dbb-990fb2c03bff","Type":"ContainerDied","Data":"007414e9c8c0a333ee0d22626d6d72ba50ccb539c7307ce4771f1656df4c3de1"}
Oct 29 23:01:01.264666 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.264649    4839 scope.go:122] "RemoveContainer" containerID="cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e"
Oct 29 23:01:01.264976 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:01.264956    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e\": container with ID starting with cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e not found: ID does not exist" containerID="cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e"
Oct 29 23:01:01.265031 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.264986    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e"} err="failed to get container status \"cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e\": rpc error: code = NotFound desc = could not find container \"cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e\": container with ID starting with cdc71a3d69e1b4a60b4f35496fdad26cda142a26c3968b15cf1fd4809256cf9e not found: ID does not exist"
Oct 29 23:01:01.265031 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.265003    4839 scope.go:122] "RemoveContainer" containerID="047c57715356baaf3fea68214c81703c1b931e141a9b1060ae6f8b4a94ff1a56"
Oct 29 23:01:01.328514 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.328484    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"]
Oct 29 23:01:01.390303 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.389045    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:01.392029 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.391999    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-a2e7c8-6d75bff575-5dfl5"]
Oct 29 23:01:01.431386 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.431339    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"]
Oct 29 23:01:01.440629 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:01.440587    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-a2e7c8-6954674ccc-qw56v"]
Oct 29 23:01:02.248890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.248853    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main" containerID="cri-o://f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252" gracePeriod=45
Oct 29 23:01:02.271378 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.269991    4839 generic.go:356] "Generic (PLEG): container finished" podID="5943fc94-a232-47b8-9efb-01b669ed2787" containerID="94a6a56bf5c5c49c09c671dcf701dbb80ea61b523b9dcbca4fc368b91cf9b978" exitCode=0
Oct 29 23:01:02.271378 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.270070    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" event={"ID":"5943fc94-a232-47b8-9efb-01b669ed2787","Type":"ContainerDied","Data":"94a6a56bf5c5c49c09c671dcf701dbb80ea61b523b9dcbca4fc368b91cf9b978"}
Oct 29 23:01:02.380211 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.380186    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 23:01:02.430965 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.430588    4839 patch_prober.go:29] interesting pod/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:02.430965 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.430691    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:02.436859 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.436435    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5943fc94-a232-47b8-9efb-01b669ed2787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943fc94-a232-47b8-9efb-01b669ed2787-plugins-config-volume\") pod \"5943fc94-a232-47b8-9efb-01b669ed2787\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") "
Oct 29 23:01:02.437732 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.437294    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5943fc94-a232-47b8-9efb-01b669ed2787-kube-api-access-xsxvm\" (UniqueName: \"kubernetes.io/projected/5943fc94-a232-47b8-9efb-01b669ed2787-kube-api-access-xsxvm\") pod \"5943fc94-a232-47b8-9efb-01b669ed2787\" (UID: \"5943fc94-a232-47b8-9efb-01b669ed2787\") "
Oct 29 23:01:02.438948 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.438901    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5943fc94-a232-47b8-9efb-01b669ed2787-plugins-config-volume" pod "5943fc94-a232-47b8-9efb-01b669ed2787" (UID: "5943fc94-a232-47b8-9efb-01b669ed2787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:02.440933 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.440682    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5943fc94-a232-47b8-9efb-01b669ed2787-kube-api-access-xsxvm" pod "5943fc94-a232-47b8-9efb-01b669ed2787" (UID: "5943fc94-a232-47b8-9efb-01b669ed2787"). InnerVolumeSpecName "kube-api-access-xsxvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.499342 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.499315    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 23:01:02.538910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.538871    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tmp-dir\") pod \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") "
Oct 29 23:01:02.539064 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539005    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-home\") pod \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") "
Oct 29 23:01:02.539064 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539055    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tls-certs\") pod \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") "
Oct 29 23:01:02.539137 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539084    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-model-cache\") pod \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") "
Oct 29 23:01:02.539137 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539085    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tmp-dir" pod "cd1ec599-e3bc-40f4-bc65-ab513668d68b" (UID: "cd1ec599-e3bc-40f4-bc65-ab513668d68b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.539258 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539189    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd1ec599-e3bc-40f4-bc65-ab513668d68b-kube-api-access-djvlt\" (UniqueName: \"kubernetes.io/projected/cd1ec599-e3bc-40f4-bc65-ab513668d68b-kube-api-access-djvlt\") pod \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") "
Oct 29 23:01:02.539258 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539225    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-dshm\") pod \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\" (UID: \"cd1ec599-e3bc-40f4-bc65-ab513668d68b\") "
Oct 29 23:01:02.539355 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539263    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-home" pod "cd1ec599-e3bc-40f4-bc65-ab513668d68b" (UID: "cd1ec599-e3bc-40f4-bc65-ab513668d68b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.539430 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539408    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-model-cache" pod "cd1ec599-e3bc-40f4-bc65-ab513668d68b" (UID: "cd1ec599-e3bc-40f4-bc65-ab513668d68b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.539683 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539663    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5943fc94-a232-47b8-9efb-01b669ed2787-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.539683 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539682    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.539866 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539693    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.539866 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539705    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.539866 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.539718    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxvm\" (UniqueName: \"kubernetes.io/projected/5943fc94-a232-47b8-9efb-01b669ed2787-kube-api-access-xsxvm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.541339 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.541302    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-dshm" pod "cd1ec599-e3bc-40f4-bc65-ab513668d68b" (UID: "cd1ec599-e3bc-40f4-bc65-ab513668d68b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:02.541465 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.541441    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tls-certs" pod "cd1ec599-e3bc-40f4-bc65-ab513668d68b" (UID: "cd1ec599-e3bc-40f4-bc65-ab513668d68b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:01:02.541465 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.541456    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd1ec599-e3bc-40f4-bc65-ab513668d68b-kube-api-access-djvlt" pod "cd1ec599-e3bc-40f4-bc65-ab513668d68b" (UID: "cd1ec599-e3bc-40f4-bc65-ab513668d68b"). InnerVolumeSpecName "kube-api-access-djvlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:02.641562 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.641481    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvlt\" (UniqueName: \"kubernetes.io/projected/cd1ec599-e3bc-40f4-bc65-ab513668d68b-kube-api-access-djvlt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.641562 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.641514    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd1ec599-e3bc-40f4-bc65-ab513668d68b-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:02.641562 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:02.641524    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd1ec599-e3bc-40f4-bc65-ab513668d68b-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.008873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.008843    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" path="/var/lib/kubelet/pods/a48caf36-726a-4a6a-8dbb-990fb2c03bff/volumes"
Oct 29 23:01:03.009273 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.009258    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8454cc6-c9b0-4a21-a803-7088233ace04" path="/var/lib/kubelet/pods/a8454cc6-c9b0-4a21-a803-7088233ace04/volumes"
Oct 29 23:01:03.208960 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.208892    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 23:01:03.241283 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:03.241239    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:01:03.248729 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.248704    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-plugins-config-volume\") pod \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") "
Oct 29 23:01:03.248848 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.248753    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-kube-api-access-k28h9\" (UniqueName: \"kubernetes.io/projected/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-kube-api-access-k28h9\") pod \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\" (UID: \"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d\") "
Oct 29 23:01:03.249123 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.249094    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-plugins-config-volume" pod "8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" (UID: "8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:03.249294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.249280    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.250910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.250890    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-kube-api-access-k28h9" pod "8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" (UID: "8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d"). InnerVolumeSpecName "kube-api-access-k28h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.283545 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.283516    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"
Oct 29 23:01:03.283965 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.283514    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9" event={"ID":"5943fc94-a232-47b8-9efb-01b669ed2787","Type":"ContainerDied","Data":"676d13e4d56ed5a12094ee79e1e63d22c01ec01162a3abb6a784357a05ab6fae"}
Oct 29 23:01:03.283965 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.283674    4839 scope.go:122] "RemoveContainer" containerID="94a6a56bf5c5c49c09c671dcf701dbb80ea61b523b9dcbca4fc368b91cf9b978"
Oct 29 23:01:03.286795 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.286367    4839 generic.go:356] "Generic (PLEG): container finished" podID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" containerID="3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7" exitCode=0
Oct 29 23:01:03.286795 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.286466    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"
Oct 29 23:01:03.286795 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.286465    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" event={"ID":"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d","Type":"ContainerDied","Data":"3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7"}
Oct 29 23:01:03.286795 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.286575    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m" event={"ID":"8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d","Type":"ContainerDied","Data":"fefc84466feb334c271de96480d45f0e740758436a23562d9f8df61a262e8267"}
Oct 29 23:01:03.288521 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.288492    4839 generic.go:356] "Generic (PLEG): container finished" podID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerID="f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252" exitCode=0
Oct 29 23:01:03.288724 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.288692    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" event={"ID":"cd1ec599-e3bc-40f4-bc65-ab513668d68b","Type":"ContainerDied","Data":"f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252"}
Oct 29 23:01:03.288849 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.288729    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf" event={"ID":"cd1ec599-e3bc-40f4-bc65-ab513668d68b","Type":"ContainerDied","Data":"3abe23d7b6cb6178149f828cda15f1a4a018a61aebf0b7d42e620464686f7f65"}
Oct 29 23:01:03.288849 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.288702    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"
Oct 29 23:01:03.299075 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.299046    4839 scope.go:122] "RemoveContainer" containerID="3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7"
Oct 29 23:01:03.310902 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.310876    4839 scope.go:122] "RemoveContainer" containerID="3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7"
Oct 29 23:01:03.311258 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:03.311220    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7\": container with ID starting with 3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7 not found: ID does not exist" containerID="3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7"
Oct 29 23:01:03.311364 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.311255    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7"} err="failed to get container status \"3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7\": rpc error: code = NotFound desc = could not find container \"3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7\": container with ID starting with 3befd24c24bd1d14db8d3c026f2439e645fae3eeac13fbdb46b95c256a2042f7 not found: ID does not exist"
Oct 29 23:01:03.311364 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.311278    4839 scope.go:122] "RemoveContainer" containerID="f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252"
Oct 29 23:01:03.321341 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.321302    4839 scope.go:122] "RemoveContainer" containerID="f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252"
Oct 29 23:01:03.321640 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:03.321619    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252\": container with ID starting with f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252 not found: ID does not exist" containerID="f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252"
Oct 29 23:01:03.321708 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.321647    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252"} err="failed to get container status \"f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252\": rpc error: code = NotFound desc = could not find container \"f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252\": container with ID starting with f8a260a8879469595402af7931c0161f702cb5cfdbacddd18ed79294a6559252 not found: ID does not exist"
Oct 29 23:01:03.350992 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.350964    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k28h9\" (UniqueName: \"kubernetes.io/projected/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d-kube-api-access-k28h9\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.368887 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.368861    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"]
Oct 29 23:01:03.374804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.374780    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-81f92d-7f994f5679-sb87m"]
Oct 29 23:01:03.479418 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.478595    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"]
Oct 29 23:01:03.520502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.493855    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-07eef2/premium-w3-07eef2-kserve-75b97fcc66-j9xqf"]
Oct 29 23:01:03.523396 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.521211    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:03.528290 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.528249    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"]
Oct 29 23:01:03.532514 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.532483    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-20f88f-59f7f6c6cf-4s7t9"]
Oct 29 23:01:03.628538 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.628515    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 23:01:03.659691 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659663    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-credential-socket\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.659848 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659703    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-certs\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.659848 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659731    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-podinfo\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.659848 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659759    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-token\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-token\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.659848 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659790    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-kube-api-access-5b9dm\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-kube-api-access-5b9dm\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.659848 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659826    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74110f0b-b412-42f2-884e-0438f0fc02a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74110f0b-b412-42f2-884e-0438f0fc02a1-istiod-ca-cert\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.660131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659862    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-socket\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.660131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659945    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-credential-socket" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.660131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.659988    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-data\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.660131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660014    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-certs" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.660131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660029    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-envoy\") pod \"74110f0b-b412-42f2-884e-0438f0fc02a1\" (UID: \"74110f0b-b412-42f2-884e-0438f0fc02a1\") "
Oct 29 23:01:03.660427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660245    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-data" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.660427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660289    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-socket" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.660427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660310    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74110f0b-b412-42f2-884e-0438f0fc02a1-istiod-ca-cert" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:03.660694 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660674    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.660781 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660694    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.660781 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660709    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74110f0b-b412-42f2-884e-0438f0fc02a1-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.660781 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660718    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.660781 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.660728    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.662076 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.662047    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-podinfo" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:03.662302 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.662284    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-envoy" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:03.662629 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.662610    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-kube-api-access-5b9dm" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "kube-api-access-5b9dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.662673 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.662638    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-token" pod "74110f0b-b412-42f2-884e-0438f0fc02a1" (UID: "74110f0b-b412-42f2-884e-0438f0fc02a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:03.762294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.762217    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.762294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.762243    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.762294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.762256    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b9dm\" (UniqueName: \"kubernetes.io/projected/74110f0b-b412-42f2-884e-0438f0fc02a1-kube-api-access-5b9dm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:03.762294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:03.762272    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74110f0b-b412-42f2-884e-0438f0fc02a1-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:04.298097 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.298042    4839 generic.go:356] "Generic (PLEG): container finished" podID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerID="da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f" exitCode=0
Oct 29 23:01:04.298591 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.298337    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"
Oct 29 23:01:04.299099 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.299072    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" event={"ID":"74110f0b-b412-42f2-884e-0438f0fc02a1","Type":"ContainerDied","Data":"da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f"}
Oct 29 23:01:04.299261 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.299124    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc" event={"ID":"74110f0b-b412-42f2-884e-0438f0fc02a1","Type":"ContainerDied","Data":"737d816448920e67c9d0d057767a68f74a9e7b8868185231c6dd0b47cdcff185"}
Oct 29 23:01:04.299261 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.299158    4839 scope.go:122] "RemoveContainer" containerID="da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f"
Oct 29 23:01:04.321740 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.321713    4839 scope.go:122] "RemoveContainer" containerID="da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f"
Oct 29 23:01:04.322051 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:04.322030    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f\": container with ID starting with da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f not found: ID does not exist" containerID="da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f"
Oct 29 23:01:04.322095 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.322062    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f"} err="failed to get container status \"da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f\": rpc error: code = NotFound desc = could not find container \"da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f\": container with ID starting with da311b3d216e484b239c76bece8ca414bd294182a53a944d551a66066443ea3f not found: ID does not exist"
Oct 29 23:01:04.369918 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.369842    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"]
Oct 29 23:01:04.383901 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:04.383871    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bf55a-openshift-default-84668b5c9-49hwc"]
Oct 29 23:01:05.009394 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.009359    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5943fc94-a232-47b8-9efb-01b669ed2787" path="/var/lib/kubelet/pods/5943fc94-a232-47b8-9efb-01b669ed2787/volumes"
Oct 29 23:01:05.009847 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.009834    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" path="/var/lib/kubelet/pods/74110f0b-b412-42f2-884e-0438f0fc02a1/volumes"
Oct 29 23:01:05.010204 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.010191    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" path="/var/lib/kubelet/pods/8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d/volumes"
Oct 29 23:01:05.010907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.010890    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" path="/var/lib/kubelet/pods/cd1ec599-e3bc-40f4-bc65-ab513668d68b/volumes"
Oct 29 23:01:05.340746 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.340708    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:05.816834 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.816803    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"]
Oct 29 23:01:05.817053 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.817029    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy" containerID="cri-o://65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9" gracePeriod=30
Oct 29 23:01:05.833126 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.833097    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"]
Oct 29 23:01:05.833347 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.833325    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy" containerID="cri-o://43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b" gracePeriod=30
Oct 29 23:01:05.862010 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.861978    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"]
Oct 29 23:01:05.862191 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.862169    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy" containerID="cri-o://62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66" gracePeriod=30
Oct 29 23:01:05.905902 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:05.905864    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:06.211375 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:06.211292    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:06.483189 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:06.483092    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:06.750533 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:06.750448    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:08.560163 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:08.560107    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:08.903011 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:08.902925    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:09.108578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.108541    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:09.458684 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.458652    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"]
Oct 29 23:01:09.458907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.458862    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" containerID="cri-o://0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90" gracePeriod=30
Oct 29 23:01:09.514169 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.510995    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"]
Oct 29 23:01:09.514169 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.511307    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" containerID="cri-o://25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5" gracePeriod=30
Oct 29 23:01:09.523604 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.523140    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"]
Oct 29 23:01:09.566222 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.530358    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy" containerID="cri-o://683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03" gracePeriod=30
Oct 29 23:01:09.595387 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.594317    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"]
Oct 29 23:01:09.595387 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.594589    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy" containerID="cri-o://7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59" gracePeriod=30
Oct 29 23:01:09.671093 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:09.671024    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:10.955603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:10.955571    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:11.226841 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.226806    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305058    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-kube-api-access-4zcxg\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-kube-api-access-4zcxg\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305182    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-credential-socket\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305240    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-istio-token\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-istio-token\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305336    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-envoy\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305371    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-data\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305439    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-certs\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305522    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7929816-c979-45d3-8dda-4688e78bc292-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7929816-c979-45d3-8dda-4688e78bc292-istiod-ca-cert\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305594    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7929816-c979-45d3-8dda-4688e78bc292-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7929816-c979-45d3-8dda-4688e78bc292-istio-podinfo\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.305649    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-socket\") pod \"d7929816-c979-45d3-8dda-4688e78bc292\" (UID: \"d7929816-c979-45d3-8dda-4688e78bc292\") "
Oct 29 23:01:11.307216 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.307068    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-credential-socket" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.307899 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.307338    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-data" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.307899 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.307869    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7929816-c979-45d3-8dda-4688e78bc292-istiod-ca-cert" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.308013 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.307942    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-certs" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.308221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.308185    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-socket" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.311570 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.311496    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:11.319619 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.315180    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-envoy" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.319619 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.318560    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7929816-c979-45d3-8dda-4688e78bc292-istio-podinfo" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.323345 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.323312    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-kube-api-access-4zcxg" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "kube-api-access-4zcxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.323471 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.323360    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-istio-token" pod "d7929816-c979-45d3-8dda-4688e78bc292" (UID: "d7929816-c979-45d3-8dda-4688e78bc292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.338751 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.338724    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 23:01:11.356442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.356369    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 23:01:11.368049 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.368020    4839 generic.go:356] "Generic (PLEG): container finished" podID="d7929816-c979-45d3-8dda-4688e78bc292" containerID="65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9" exitCode=0
Oct 29 23:01:11.368240 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.368064    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"
Oct 29 23:01:11.368240 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.368106    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" event={"ID":"d7929816-c979-45d3-8dda-4688e78bc292","Type":"ContainerDied","Data":"65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9"}
Oct 29 23:01:11.368240 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.368187    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4" event={"ID":"d7929816-c979-45d3-8dda-4688e78bc292","Type":"ContainerDied","Data":"a986a489f016131c72b2b82366d4d2f217b5c0f6151a12db27d5630b0e1964e4"}
Oct 29 23:01:11.368240 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.368219    4839 scope.go:122] "RemoveContainer" containerID="65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9"
Oct 29 23:01:11.369764 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.369740    4839 generic.go:356] "Generic (PLEG): container finished" podID="10f4df42-81b7-4660-82b4-04c567edc874" containerID="62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66" exitCode=0
Oct 29 23:01:11.369898 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.369831    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" event={"ID":"10f4df42-81b7-4660-82b4-04c567edc874","Type":"ContainerDied","Data":"62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66"}
Oct 29 23:01:11.369898 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.369887    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t" event={"ID":"10f4df42-81b7-4660-82b4-04c567edc874","Type":"ContainerDied","Data":"73a13c829657a416d8b7ca683eb42669e07981d25789b540866047087e953831"}
Oct 29 23:01:11.370092 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.370076    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"
Oct 29 23:01:11.371870 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.371844    4839 generic.go:356] "Generic (PLEG): container finished" podID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerID="43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b" exitCode=0
Oct 29 23:01:11.371995 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.371880    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"
Oct 29 23:01:11.371995 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.371913    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" event={"ID":"a41b48a5-2064-4e12-87b5-4c8b3fb8694c","Type":"ContainerDied","Data":"43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b"}
Oct 29 23:01:11.371995 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.371952    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4" event={"ID":"a41b48a5-2064-4e12-87b5-4c8b3fb8694c","Type":"ContainerDied","Data":"99a072f8d260b5fb185b6e4f2599585fa4eb1808fa18707375b72c7aa8f05c11"}
Oct 29 23:01:11.384786 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.383924    4839 scope.go:122] "RemoveContainer" containerID="65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9"
Oct 29 23:01:11.384786 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:11.384307    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9\": container with ID starting with 65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9 not found: ID does not exist" containerID="65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9"
Oct 29 23:01:11.384786 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.384335    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9"} err="failed to get container status \"65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9\": rpc error: code = NotFound desc = could not find container \"65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9\": container with ID starting with 65a62de563b7c1f7475df7f12e47d2b58036e6b334dce051ea74e6b88cdf94b9 not found: ID does not exist"
Oct 29 23:01:11.384786 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.384353    4839 scope.go:122] "RemoveContainer" containerID="62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66"
Oct 29 23:01:11.398748 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.398723    4839 scope.go:122] "RemoveContainer" containerID="62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66"
Oct 29 23:01:11.399133 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:11.399108    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66\": container with ID starting with 62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66 not found: ID does not exist" containerID="62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66"
Oct 29 23:01:11.399250 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.399160    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66"} err="failed to get container status \"62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66\": rpc error: code = NotFound desc = could not find container \"62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66\": container with ID starting with 62b6f57c690dd43220f71878b0049bb30e039c570bd7c1f4d80effce96731d66 not found: ID does not exist"
Oct 29 23:01:11.399250 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.399188    4839 scope.go:122] "RemoveContainer" containerID="43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b"
Oct 29 23:01:11.406707 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406677    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-istio-token\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-istio-token\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.406830 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406744    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-certs\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.406830 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406780    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-kube-api-access-6j984\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-kube-api-access-6j984\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.406830 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406811    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-socket\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.407024 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406840    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-envoy\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407024 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406891    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-credential-socket\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407024 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406960    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-socket\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407024 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.406998    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f4df42-81b7-4660-82b4-04c567edc874-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f4df42-81b7-4660-82b4-04c567edc874-istiod-ca-cert\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.407279 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407069    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-kube-api-access-w24kw\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-kube-api-access-w24kw\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407279 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407127    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istiod-ca-cert\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407279 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407199    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-certs\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407279 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407225    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-token\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-token\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407279 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407242    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-socket" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.407279 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407265    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-envoy\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.407588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407294    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10f4df42-81b7-4660-82b4-04c567edc874-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f4df42-81b7-4660-82b4-04c567edc874-istio-podinfo\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.407588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407319    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-certs" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.407588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407343    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-credential-socket\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.407588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407434    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-data\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407497    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-data\") pod \"10f4df42-81b7-4660-82b4-04c567edc874\" (UID: \"10f4df42-81b7-4660-82b4-04c567edc874\") "
Oct 29 23:01:11.407588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407563    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-podinfo\") pod \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\" (UID: \"a41b48a5-2064-4e12-87b5-4c8b3fb8694c\") "
Oct 29 23:01:11.407910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.407888    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-credential-socket" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.408122 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408100    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408220 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408130    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408220 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408161    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408220 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408182    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408220 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408201    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408220 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408215    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408228    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408241    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7929816-c979-45d3-8dda-4688e78bc292-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408254    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7929816-c979-45d3-8dda-4688e78bc292-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408266    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408280    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zcxg\" (UniqueName: \"kubernetes.io/projected/d7929816-c979-45d3-8dda-4688e78bc292-kube-api-access-4zcxg\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408295    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7929816-c979-45d3-8dda-4688e78bc292-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.408782 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.408758    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-socket" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.409727 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.409697    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-credential-socket" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.410128 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.410103    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f4df42-81b7-4660-82b4-04c567edc874-istiod-ca-cert" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.410346 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.410316    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-data" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.410498 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.410478    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-data" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.410660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.410636    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istiod-ca-cert" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:11.410873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.410847    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-certs" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.411300 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.411280    4839 scope.go:122] "RemoveContainer" containerID="7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862"
Oct 29 23:01:11.412278 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.412250    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10f4df42-81b7-4660-82b4-04c567edc874-istio-podinfo" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.412886 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.412859    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-token" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.414242 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.414214    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-envoy" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.414496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.414468    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-podinfo" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:11.414553 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.414508    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-kube-api-access-6j984" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "kube-api-access-6j984". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.414553 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.414530    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-istio-token" pod "10f4df42-81b7-4660-82b4-04c567edc874" (UID: "10f4df42-81b7-4660-82b4-04c567edc874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.414553 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.414543    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-kube-api-access-w24kw" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "kube-api-access-w24kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:11.415976 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.415951    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-envoy" pod "a41b48a5-2064-4e12-87b5-4c8b3fb8694c" (UID: "a41b48a5-2064-4e12-87b5-4c8b3fb8694c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:11.422324 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.422299    4839 scope.go:122] "RemoveContainer" containerID="43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b"
Oct 29 23:01:11.422607 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:11.422583    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b\": container with ID starting with 43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b not found: ID does not exist" containerID="43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b"
Oct 29 23:01:11.422658 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.422616    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b"} err="failed to get container status \"43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b\": rpc error: code = NotFound desc = could not find container \"43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b\": container with ID starting with 43462fe5a0f08fea5d274324104225dbfa02525cb5377d78264bd5bb051b738b not found: ID does not exist"
Oct 29 23:01:11.422658 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.422633    4839 scope.go:122] "RemoveContainer" containerID="7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862"
Oct 29 23:01:11.422898 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:11.422878    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862\": container with ID starting with 7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862 not found: ID does not exist" containerID="7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862"
Oct 29 23:01:11.422947 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.422904    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862"} err="failed to get container status \"7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862\": rpc error: code = NotFound desc = could not find container \"7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862\": container with ID starting with 7f6671b611bc459e4aea406ebfd28beea5f2be3e0168250efbcd626e1c530862 not found: ID does not exist"
Oct 29 23:01:11.509355 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509320    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509355 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509351    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10f4df42-81b7-4660-82b4-04c567edc874-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509355 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509363    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w24kw\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-kube-api-access-w24kw\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509374    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509383    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509390    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509398    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509405    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10f4df42-81b7-4660-82b4-04c567edc874-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509413    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509421    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509428    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10f4df42-81b7-4660-82b4-04c567edc874-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509438    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509445    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509453    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j984\" (UniqueName: \"kubernetes.io/projected/10f4df42-81b7-4660-82b4-04c567edc874-kube-api-access-6j984\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.509598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.509463    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a41b48a5-2064-4e12-87b5-4c8b3fb8694c-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:11.615111 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.615025    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"]
Oct 29 23:01:11.630567 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.630533    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:11.638999 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.638952    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-dmtc4"]
Oct 29 23:01:11.761952 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.761920    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"]
Oct 29 23:01:11.807402 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.807365    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-pbq9t"]
Oct 29 23:01:11.861387 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.861353    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"]
Oct 29 23:01:11.884352 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:11.884277    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-81f92d-openshift-default-9cbc48765-tfnp4"]
Oct 29 23:01:12.483765 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:12.483728    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:12.484188 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:12.483812    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:12.711004 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:12.710964    4839 patch_prober.go:29] interesting pod/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:12.711194 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:12.711033    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:13.008842 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:13.008806    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f4df42-81b7-4660-82b4-04c567edc874" path="/var/lib/kubelet/pods/10f4df42-81b7-4660-82b4-04c567edc874/volumes"
Oct 29 23:01:13.009316 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:13.009296    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" path="/var/lib/kubelet/pods/a41b48a5-2064-4e12-87b5-4c8b3fb8694c/volumes"
Oct 29 23:01:13.009722 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:13.009710    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7929816-c979-45d3-8dda-4688e78bc292" path="/var/lib/kubelet/pods/d7929816-c979-45d3-8dda-4688e78bc292/volumes"
Oct 29 23:01:13.546469 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:13.546437    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:14.271324 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.271296    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:14.458538 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.458506    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"]
Oct 29 23:01:14.470764 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.470727    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-07eef2-dg2zz"]
Oct 29 23:01:14.806298 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.806264    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 23:01:14.851270 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.851240    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:14.883703 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.883619    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-kube-api-access-6pn82\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-kube-api-access-6pn82\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884030 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.883986    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-socket\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884281 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.884258    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-podinfo\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884455 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.884338    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-credential-socket\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884455 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.884379    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-token\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-token\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884455 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.884415    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-certs\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884631 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.884523    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-envoy\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884631 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.884565    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-data\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.884631 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.884594    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istiod-ca-cert\") pod \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\" (UID: \"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f\") "
Oct 29 23:01:14.885467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.885439    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istiod-ca-cert" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:14.886342 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.886276    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-credential-socket" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.886494 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.886466    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-socket" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.886621 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.886580    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-data" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.886621 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.886593    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-certs" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.887383 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.887354    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-kube-api-access-6pn82" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "kube-api-access-6pn82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.888979 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.888944    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-envoy" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.889074 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.889036    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 23:01:14.889326 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.889263    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-token" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.889453 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.889427    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-podinfo" pod "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" (UID: "1472f1e3-2edc-4ce5-9d92-4603e6bfc87f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:14.941788 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.941755    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 23:01:14.951512 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.951489    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 23:01:14.985498 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985462    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-podinfo\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.985686 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985511    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-socket\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.985686 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985581    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-credential-socket\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.985686 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985652    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a28d4a2f-4950-471a-8de9-bba72662c7b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d4a2f-4950-471a-8de9-bba72662c7b8-istiod-ca-cert\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.985884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985705    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-token\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-token\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.985884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985756    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-certs\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.985884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985807    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-data\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.985884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985836    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-kube-api-access-h49wl\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-kube-api-access-h49wl\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.986096 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.985926    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-envoy\") pod \"a28d4a2f-4950-471a-8de9-bba72662c7b8\" (UID: \"a28d4a2f-4950-471a-8de9-bba72662c7b8\") "
Oct 29 23:01:14.986218 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986194    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-certs" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986088    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-socket" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986463    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986485    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986502    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986519    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986533    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986545    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986557    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pn82\" (UniqueName: \"kubernetes.io/projected/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-kube-api-access-6pn82\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986571    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986583    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986595    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:14.987175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.986847    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-data" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.988091 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.987822    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-credential-socket" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:14.988423 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.988384    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28d4a2f-4950-471a-8de9-bba72662c7b8-istiod-ca-cert" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:14.988994 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.988958    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-podinfo" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:14.989867 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.989844    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-kube-api-access-h49wl" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "kube-api-access-h49wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.990305 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.990278    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-token" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:14.990583 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:14.990555    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-envoy" pod "a28d4a2f-4950-471a-8de9-bba72662c7b8" (UID: "a28d4a2f-4950-471a-8de9-bba72662c7b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.009037 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.009004    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e35069-c61b-4d56-b47e-70b08f54dffe" path="/var/lib/kubelet/pods/34e35069-c61b-4d56-b47e-70b08f54dffe/volumes"
Oct 29 23:01:15.087215 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087180    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-socket\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.087379 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087239    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-kube-api-access-pktk5\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-kube-api-access-pktk5\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.087379 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087289    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-socket\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.087379 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087325    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-token\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-token\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.087496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087387    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-certs\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.087496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087426    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-podinfo\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.087496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087467    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-credential-socket\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.087496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087474    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-socket" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.087696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087496    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istiod-ca-cert\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.087696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087540    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-data\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.087696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087566    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2821795-ad5a-4f7c-b0ea-080a8437f717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2821795-ad5a-4f7c-b0ea-080a8437f717-istiod-ca-cert\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.087696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087612    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-envoy\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.087696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087606    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-socket" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.087696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087643    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-podinfo\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.087696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087678    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-certs" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.088061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087766    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-kube-api-access-vccqv\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-kube-api-access-vccqv\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.088061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087874    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-envoy\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.088061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087902    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-credential-socket" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.088061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087935    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-credential-socket\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.088061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087971    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-data\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.088061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.087994    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2821795-ad5a-4f7c-b0ea-080a8437f717-istiod-ca-cert" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:15.088061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088002    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-certs\") pod \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\" (UID: \"68408cd1-eda7-4df5-80ad-e82f9cf809e9\") "
Oct 29 23:01:15.088483 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088115    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-data" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.088483 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088160    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-token\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-token\") pod \"d2821795-ad5a-4f7c-b0ea-080a8437f717\" (UID: \"d2821795-ad5a-4f7c-b0ea-080a8437f717\") "
Oct 29 23:01:15.088483 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088201    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-certs" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.088813 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088788    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.088884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088820    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.088884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088835    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.088884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088850    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.088884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088863    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2821795-ad5a-4f7c-b0ea-080a8437f717-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.088884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088876    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088890    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h49wl\" (UniqueName: \"kubernetes.io/projected/a28d4a2f-4950-471a-8de9-bba72662c7b8-kube-api-access-h49wl\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088905    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088918    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d4a2f-4950-471a-8de9-bba72662c7b8-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088932    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088946    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088959    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088973    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d4a2f-4950-471a-8de9-bba72662c7b8-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.088986    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.089000    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d4a2f-4950-471a-8de9-bba72662c7b8-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.089886 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.089856    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istiod-ca-cert" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:15.090226 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090136    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-token" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:15.090226 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090211    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-podinfo" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:15.090394 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090220    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-credential-socket" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.090478 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090447    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-podinfo" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:15.090588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090486    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-data" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.090659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090601    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-kube-api-access-pktk5" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "kube-api-access-pktk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:15.090717 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090694    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-kube-api-access-vccqv" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "kube-api-access-vccqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:15.090936 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.090915    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-token" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:15.091910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.091890    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-envoy" pod "d2821795-ad5a-4f7c-b0ea-080a8437f717" (UID: "d2821795-ad5a-4f7c-b0ea-080a8437f717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.092223 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.092186    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-envoy" pod "68408cd1-eda7-4df5-80ad-e82f9cf809e9" (UID: "68408cd1-eda7-4df5-80ad-e82f9cf809e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:15.104293 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.104267    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190179    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190214    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190224    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190232    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vccqv\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-kube-api-access-vccqv\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190243    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190253    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190266    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190278    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190290    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktk5\" (UniqueName: \"kubernetes.io/projected/d2821795-ad5a-4f7c-b0ea-080a8437f717-kube-api-access-pktk5\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190300    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68408cd1-eda7-4df5-80ad-e82f9cf809e9-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.190636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.190308    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2821795-ad5a-4f7c-b0ea-080a8437f717-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:15.417098 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.417065    4839 generic.go:356] "Generic (PLEG): container finished" podID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerID="683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03" exitCode=0
Oct 29 23:01:15.417290 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.417133    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"
Oct 29 23:01:15.417290 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.417206    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" event={"ID":"68408cd1-eda7-4df5-80ad-e82f9cf809e9","Type":"ContainerDied","Data":"683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03"}
Oct 29 23:01:15.417453 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.417317    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk" event={"ID":"68408cd1-eda7-4df5-80ad-e82f9cf809e9","Type":"ContainerDied","Data":"1db358bb08fcea0f62140463cd8bc0e346b3bd2e9ad961bb63c3b1a1c64963b6"}
Oct 29 23:01:15.417453 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.417351    4839 scope.go:122] "RemoveContainer" containerID="683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03"
Oct 29 23:01:15.418753 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.418730    4839 generic.go:356] "Generic (PLEG): container finished" podID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerID="0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90" exitCode=0
Oct 29 23:01:15.419225 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.418754    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" event={"ID":"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f","Type":"ContainerDied","Data":"0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90"}
Oct 29 23:01:15.419225 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.418789    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"
Oct 29 23:01:15.419225 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.418805    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n" event={"ID":"1472f1e3-2edc-4ce5-9d92-4603e6bfc87f","Type":"ContainerDied","Data":"ef7b9208c4a83bf01c434fafb263570da30551a55b97392703fd45728dd4de6e"}
Oct 29 23:01:15.420565 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.420538    4839 generic.go:356] "Generic (PLEG): container finished" podID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerID="25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5" exitCode=0
Oct 29 23:01:15.420963 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.420603    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"
Oct 29 23:01:15.420963 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.420626    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" event={"ID":"a28d4a2f-4950-471a-8de9-bba72662c7b8","Type":"ContainerDied","Data":"25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5"}
Oct 29 23:01:15.420963 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.420675    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf" event={"ID":"a28d4a2f-4950-471a-8de9-bba72662c7b8","Type":"ContainerDied","Data":"e62612923a661effbdefb17460f80db0b17208372a949e088a2a68220299467d"}
Oct 29 23:01:15.422773 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.422752    4839 generic.go:356] "Generic (PLEG): container finished" podID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerID="7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59" exitCode=0
Oct 29 23:01:15.422938 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.422821    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"
Oct 29 23:01:15.422938 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.422858    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" event={"ID":"d2821795-ad5a-4f7c-b0ea-080a8437f717","Type":"ContainerDied","Data":"7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59"}
Oct 29 23:01:15.422938 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.422903    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n" event={"ID":"d2821795-ad5a-4f7c-b0ea-080a8437f717","Type":"ContainerDied","Data":"34113bb7c2fcf974c985c428b2b1ec1a8483c95a337215a9eec7824309295868"}
Oct 29 23:01:15.429443 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.429422    4839 scope.go:122] "RemoveContainer" containerID="f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796"
Oct 29 23:01:15.440130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.440115    4839 scope.go:122] "RemoveContainer" containerID="683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03"
Oct 29 23:01:15.440523 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:15.440464    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03\": container with ID starting with 683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03 not found: ID does not exist" containerID="683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03"
Oct 29 23:01:15.440523 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.440502    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03"} err="failed to get container status \"683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03\": rpc error: code = NotFound desc = could not find container \"683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03\": container with ID starting with 683090f3ad32627c170a669e632083682a370e14c9763b1af02eb455d2705f03 not found: ID does not exist"
Oct 29 23:01:15.440695 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.440527    4839 scope.go:122] "RemoveContainer" containerID="f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796"
Oct 29 23:01:15.440821 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:15.440803    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796\": container with ID starting with f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796 not found: ID does not exist" containerID="f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796"
Oct 29 23:01:15.440890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.440830    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796"} err="failed to get container status \"f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796\": rpc error: code = NotFound desc = could not find container \"f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796\": container with ID starting with f28c6d26060e21bbbda98003eb4eb898701eb96b3582b651f89c6e39be2b2796 not found: ID does not exist"
Oct 29 23:01:15.440890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.440851    4839 scope.go:122] "RemoveContainer" containerID="0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90"
Oct 29 23:01:15.451117 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.451094    4839 scope.go:122] "RemoveContainer" containerID="0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90"
Oct 29 23:01:15.451448 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:15.451426    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90\": container with ID starting with 0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90 not found: ID does not exist" containerID="0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90"
Oct 29 23:01:15.451502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.451456    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90"} err="failed to get container status \"0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90\": rpc error: code = NotFound desc = could not find container \"0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90\": container with ID starting with 0312199fe3e75df100334bbc5aac1ffa2a64232c711527300e6bbf2e8d3ede90 not found: ID does not exist"
Oct 29 23:01:15.451502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.451475    4839 scope.go:122] "RemoveContainer" containerID="25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5"
Oct 29 23:01:15.460476 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.460457    4839 scope.go:122] "RemoveContainer" containerID="25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5"
Oct 29 23:01:15.460738 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:15.460719    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5\": container with ID starting with 25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5 not found: ID does not exist" containerID="25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5"
Oct 29 23:01:15.460780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.460750    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5"} err="failed to get container status \"25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5\": rpc error: code = NotFound desc = could not find container \"25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5\": container with ID starting with 25ddb79baa8991abb974ba042a074035a1190950084cfb41fd4ab27ddee8c9c5 not found: ID does not exist"
Oct 29 23:01:15.460780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.460769    4839 scope.go:122] "RemoveContainer" containerID="7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59"
Oct 29 23:01:15.469817 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.469800    4839 scope.go:122] "RemoveContainer" containerID="7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59"
Oct 29 23:01:15.470135 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:15.470114    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59\": container with ID starting with 7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59 not found: ID does not exist" containerID="7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59"
Oct 29 23:01:15.470233 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.470157    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59"} err="failed to get container status \"7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59\": rpc error: code = NotFound desc = could not find container \"7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59\": container with ID starting with 7910b52ea737564a21aea4ca30c3a4db7ac93a6aeab6a741db1739a91a72eb59 not found: ID does not exist"
Oct 29 23:01:15.555454 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.555417    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"]
Oct 29 23:01:15.569187 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.569135    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-mkp8n"]
Oct 29 23:01:15.576612 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.576582    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:15.627340 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.627309    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"]
Oct 29 23:01:15.658282 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.658252    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-cnxrk"]
Oct 29 23:01:15.705597 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.705512    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"]
Oct 29 23:01:15.715667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.715631    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-vh7vf"]
Oct 29 23:01:15.817052 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.817019    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"]
Oct 29 23:01:15.838844 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.838813    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-a2e7c8-openshift-default-694c67bc49-5fj8n"]
Oct 29 23:01:15.996379 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:15.996305    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:16.127680 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:16.127642    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"]
Oct 29 23:01:16.128289 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:16.127857    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" containerID="cri-o://27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a" gracePeriod=30
Oct 29 23:01:16.179698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:16.179667    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"]
Oct 29 23:01:16.179936 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:16.179911    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy" containerID="cri-o://ca7b1b7684c70d93401669d78986137a24fabecae00c37f00222be2076b88d40" gracePeriod=30
Oct 29 23:01:16.359301 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:16.359224    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:17.009066 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:17.009033    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" path="/var/lib/kubelet/pods/1472f1e3-2edc-4ce5-9d92-4603e6bfc87f/volumes"
Oct 29 23:01:17.009544 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:17.009528    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" path="/var/lib/kubelet/pods/68408cd1-eda7-4df5-80ad-e82f9cf809e9/volumes"
Oct 29 23:01:17.009944 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:17.009932    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" path="/var/lib/kubelet/pods/a28d4a2f-4950-471a-8de9-bba72662c7b8/volumes"
Oct 29 23:01:17.010660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:17.010643    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" path="/var/lib/kubelet/pods/d2821795-ad5a-4f7c-b0ea-080a8437f717/volumes"
Oct 29 23:01:17.652087 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:17.652050    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:18.322442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:18.322413    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:19.400518 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:19.396027    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:19.702552 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:19.702475    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:19.943884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:19.943848    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:20.165058 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:20.165029    4839 patch_prober.go:29] interesting pod/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:01:20.165245 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:20.165085    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:01:20.733134 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:20.733097    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:21.438506 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.438479    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 23:01:21.475992 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.475963    4839 generic.go:356] "Generic (PLEG): container finished" podID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerID="27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a" exitCode=0
Oct 29 23:01:21.476126 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.476015    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"
Oct 29 23:01:21.476126 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.476043    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" event={"ID":"65f5a414-18f6-4063-9e8a-62d7cc7913dd","Type":"ContainerDied","Data":"27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a"}
Oct 29 23:01:21.476126 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.476099    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2" event={"ID":"65f5a414-18f6-4063-9e8a-62d7cc7913dd","Type":"ContainerDied","Data":"7c08d113745c3d58bf8037b82d31399dda75dd99e9957f58e56f688de29acebf"}
Oct 29 23:01:21.476126 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.476123    4839 scope.go:122] "RemoveContainer" containerID="27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a"
Oct 29 23:01:21.478286 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.478262    4839 generic.go:356] "Generic (PLEG): container finished" podID="11c17881-8e31-4bc4-a55e-6005c5225912" containerID="ca7b1b7684c70d93401669d78986137a24fabecae00c37f00222be2076b88d40" exitCode=0
Oct 29 23:01:21.478401 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.478333    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" event={"ID":"11c17881-8e31-4bc4-a55e-6005c5225912","Type":"ContainerDied","Data":"ca7b1b7684c70d93401669d78986137a24fabecae00c37f00222be2076b88d40"}
Oct 29 23:01:21.478401 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.478374    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr" event={"ID":"11c17881-8e31-4bc4-a55e-6005c5225912","Type":"ContainerDied","Data":"5554a09aafea1d22886510f939b58e4d4935265f6356285deab9ea34a582bd9b"}
Oct 29 23:01:21.478401 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.478398    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5554a09aafea1d22886510f939b58e4d4935265f6356285deab9ea34a582bd9b"
Oct 29 23:01:21.487377 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.487353    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 23:01:21.488739 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.488726    4839 scope.go:122] "RemoveContainer" containerID="27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a"
Oct 29 23:01:21.488999 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:21.488981    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a\": container with ID starting with 27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a not found: ID does not exist" containerID="27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a"
Oct 29 23:01:21.489040 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.489007    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a"} err="failed to get container status \"27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a\": rpc error: code = NotFound desc = could not find container \"27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a\": container with ID starting with 27a3d53f53b4f07802fba84338499ab16e7602492596edd32f6b23ae400fe04a not found: ID does not exist"
Oct 29 23:01:21.489040 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.489027    4839 scope.go:122] "RemoveContainer" containerID="61e15854a4450b37531bc037bf56e680feff016aa0027daa1088529d478863bd"
Oct 29 23:01:21.591936 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.591905    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-credential-socket\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.592118 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.591958    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istiod-ca-cert\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.592118 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592021    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-envoy\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.592118 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592049    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-envoy\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.592118 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592082    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-kube-api-access-g75dp\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-kube-api-access-g75dp\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.592359 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592127    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-data\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.592359 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592206    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-istio-token\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-istio-token\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.592359 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592206    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-credential-socket" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.592359 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592267    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-credential-socket\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.592564 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592408    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-data\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.592564 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592460    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-socket\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.592564 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592511    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-certs\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.592806 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592786    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-certs\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.593032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592513    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istiod-ca-cert" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:21.593032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592762    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-data" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.593032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592763    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-socket" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.593032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.593010    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-certs" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.593328 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592853    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-data" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.593328 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592869    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-certs" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.593328 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.592969    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-credential-socket" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.593559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.593536    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11c17881-8e31-4bc4-a55e-6005c5225912-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c17881-8e31-4bc4-a55e-6005c5225912-istio-podinfo\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.593688 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.593595    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-token\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-token\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.593688 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.593657    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-kube-api-access-tvj2c\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-kube-api-access-tvj2c\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.593817 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.593691    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c17881-8e31-4bc4-a55e-6005c5225912-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c17881-8e31-4bc4-a55e-6005c5225912-istiod-ca-cert\") pod \"11c17881-8e31-4bc4-a55e-6005c5225912\" (UID: \"11c17881-8e31-4bc4-a55e-6005c5225912\") "
Oct 29 23:01:21.593817 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.593732    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-podinfo\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.593817 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.593777    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-socket\") pod \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\" (UID: \"65f5a414-18f6-4063-9e8a-62d7cc7913dd\") "
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594247    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594272    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594307    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594321    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594334    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594351    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594364    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594378    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594583    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-socket" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594915    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c17881-8e31-4bc4-a55e-6005c5225912-istiod-ca-cert" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.594991    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-envoy" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.595084    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-istio-token" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.595124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.595111    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-kube-api-access-g75dp" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "kube-api-access-g75dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.595847 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.595218    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-envoy" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:21.596652 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.596626    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-podinfo" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:21.596883 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.596865    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-kube-api-access-tvj2c" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "kube-api-access-tvj2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.597942 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.597921    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11c17881-8e31-4bc4-a55e-6005c5225912-istio-podinfo" pod "11c17881-8e31-4bc4-a55e-6005c5225912" (UID: "11c17881-8e31-4bc4-a55e-6005c5225912"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:21.598121 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.598102    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-token" pod "65f5a414-18f6-4063-9e8a-62d7cc7913dd" (UID: "65f5a414-18f6-4063-9e8a-62d7cc7913dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:21.695698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695614    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11c17881-8e31-4bc4-a55e-6005c5225912-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.695698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695646    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.695698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695655    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvj2c\" (UniqueName: \"kubernetes.io/projected/65f5a414-18f6-4063-9e8a-62d7cc7913dd-kube-api-access-tvj2c\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.695698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695676    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11c17881-8e31-4bc4-a55e-6005c5225912-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.695698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695685    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.695698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695696    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.695698 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695706    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65f5a414-18f6-4063-9e8a-62d7cc7913dd-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.696028 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695714    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11c17881-8e31-4bc4-a55e-6005c5225912-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.696028 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695721    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75dp\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-kube-api-access-g75dp\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.696028 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.695729    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11c17881-8e31-4bc4-a55e-6005c5225912-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:21.843298 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.843263    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"]
Oct 29 23:01:21.850827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:21.850795    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-2nnl2"]
Oct 29 23:01:22.094543 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:22.094510    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:22.487643 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:22.487562    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"
Oct 29 23:01:22.613832 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:22.613791    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"]
Oct 29 23:01:22.621880 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:22.621835    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-20f88f-openshift-default-68c49c98f9-9hjmr"]
Oct 29 23:01:23.008444 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.008412    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" path="/var/lib/kubelet/pods/11c17881-8e31-4bc4-a55e-6005c5225912/volumes"
Oct 29 23:01:23.008875 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.008860    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" path="/var/lib/kubelet/pods/65f5a414-18f6-4063-9e8a-62d7cc7913dd/volumes"
Oct 29 23:01:23.566104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.566070    4839 scope.go:122] "RemoveContainer" containerID="1648c059da92e742ff7fb2cd154f76b0e9579d0c56ac0989a88782044fa8c1dc"
Oct 29 23:01:23.575633 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.575607    4839 scope.go:122] "RemoveContainer" containerID="b5b1edd6a5d164323ac48933d64460e8ff8b9c5fd0417205e33c2df8c86ee1ff"
Oct 29 23:01:23.584296 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.584273    4839 scope.go:122] "RemoveContainer" containerID="c8bcd49a7d7ddc15ef344e2a3707f53b912a9187789d1f863f02a0bcbef994db"
Oct 29 23:01:23.592989 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.592971    4839 scope.go:122] "RemoveContainer" containerID="fe47ec1b60f33ed788168fd893048d00d3118311206d9a775de1630e61f1b093"
Oct 29 23:01:23.601596 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.601575    4839 scope.go:122] "RemoveContainer" containerID="046d01dc7f8f05905e550910e254440f45ece7a6d8b3f255392a997ac53bd1c0"
Oct 29 23:01:23.610413 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.610390    4839 scope.go:122] "RemoveContainer" containerID="ca7b1b7684c70d93401669d78986137a24fabecae00c37f00222be2076b88d40"
Oct 29 23:01:23.619543 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.619520    4839 scope.go:122] "RemoveContainer" containerID="7ef9288b9e0afc68058a7bb3fde6c4f8fd2660161c82b2493cd41b4d863b78da"
Oct 29 23:01:23.628552 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.628529    4839 scope.go:122] "RemoveContainer" containerID="baff145962dab3e7c234b6b16dbfd2d0942496c6a3b38a6452d9a5d90a30c5ba"
Oct 29 23:01:23.638011 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:23.637988    4839 scope.go:122] "RemoveContainer" containerID="735c95c610f00b04a521d208e9ed334823caba1324f8a55003d486d1e41177df"
Oct 29 23:01:24.184200 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.182489    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.716492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.716461    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.896109 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.896073    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:24.910954 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.910923    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"]
Oct 29 23:01:24.911215 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.911141    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy" containerID="cri-o://0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5" gracePeriod=30
Oct 29 23:01:24.924819 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.924792    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"]
Oct 29 23:01:24.925052 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.925008    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy" containerID="cri-o://59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce" gracePeriod=30
Oct 29 23:01:24.949020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.948992    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"]
Oct 29 23:01:24.949225 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.949204    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy" containerID="cri-o://0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d" gracePeriod=30
Oct 29 23:01:24.962204 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.962174    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"]
Oct 29 23:01:24.962400 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:24.962378    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy" containerID="cri-o://c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404" gracePeriod=30
Oct 29 23:01:25.072420 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:25.072383    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:26.233287 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:26.232706    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:27.096111 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:27.096076    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:28.173219 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:28.173185    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:28.326573 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:28.326538    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:28.884619 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:28.884583    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:29.511796 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:29.511749    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:29.908717 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:29.908666    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:30.034471 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.034436    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"]
Oct 29 23:01:30.048134 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.048106    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f9d3a66-hdmlz"]
Oct 29 23:01:30.240169 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.240105    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 23:01:30.283114 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.283086    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 23:01:30.289236 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.289212    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 23:01:30.295398 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.292935    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 23:01:30.336493 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336461    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-token\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-token\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.336659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336519    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-socket\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.336659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336551    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-certs\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.336659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336575    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7463fcbe-ed3c-4234-828d-955fa96a394a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463fcbe-ed3c-4234-828d-955fa96a394a-istiod-ca-cert\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.336659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336602    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-token\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-token\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.336659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336626    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-envoy\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336668    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-data\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336702    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-socket\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336735    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-kube-api-access-4sqnt\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-kube-api-access-4sqnt\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336769    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-token\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-token\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336802    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-kube-api-access-9qn5b\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-kube-api-access-9qn5b\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336842    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-kube-api-access-dl59j\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-kube-api-access-dl59j\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336887    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-socket\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.336937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336918    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-data\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.337374 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336946    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-certs\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.337374 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336991    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istiod-ca-cert\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.337374 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337032    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-envoy\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.336800    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-socket" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337004    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-data" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337002    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-certs" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337241    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-socket" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337267    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-data" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337373    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-certs" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337510    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istiod-ca-cert" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:30.337603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337511    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-socket" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337633    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istiod-ca-cert\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337677    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-podinfo\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337708    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-certs\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337737    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-podinfo\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337771    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-certs\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337810    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-envoy\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337861    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istiod-ca-cert\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337922    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-envoy\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.337993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.337965    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-credential-socket\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338000    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-kube-api-access-shgk5\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-kube-api-access-shgk5\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338062    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-data\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338174    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-podinfo\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338208    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-podinfo\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338258    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-data\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338289    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-credential-socket\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338321    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-credential-socket\") pod \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\" (UID: \"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338339    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7463fcbe-ed3c-4234-828d-955fa96a394a-istiod-ca-cert" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338352    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-socket\") pod \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\" (UID: \"ad61179b-7e0c-4bde-a7ee-e8272ae300b7\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338398    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-credential-socket\") pod \"7463fcbe-ed3c-4234-828d-955fa96a394a\" (UID: \"7463fcbe-ed3c-4234-828d-955fa96a394a\") "
Oct 29 23:01:30.338495 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338428    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-token\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-token\") pod \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\" (UID: \"6cae13af-8c9f-4da1-9e48-5e22cfb39b42\") "
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338835    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istiod-ca-cert" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338913    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338937    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338952    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338965    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338978    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.338992    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.339005    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7463fcbe-ed3c-4234-828d-955fa96a394a-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.339018    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339036 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.339031    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.339641 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.339297    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-certs" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.339641 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.339568    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-data" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.339761 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.339744    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istiod-ca-cert" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:01:30.340070 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.340017    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-credential-socket" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.340243 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.340087    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-credential-socket" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.340737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.340291    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-socket" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.340737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.340353    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-certs" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.340737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.340526    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-credential-socket" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.340737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.340570    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-data" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.341034 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.340763    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-credential-socket" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.341588 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.341564    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-token" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.341706 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.341623    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-token" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.341706 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.341640    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-token" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.344956 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.344927    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-podinfo" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345545    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-podinfo" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345576    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-token" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345579    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-envoy" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345595    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-podinfo" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345605    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-podinfo" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345607    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-kube-api-access-shgk5" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "kube-api-access-shgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345623    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-kube-api-access-4sqnt" pod "6cae13af-8c9f-4da1-9e48-5e22cfb39b42" (UID: "6cae13af-8c9f-4da1-9e48-5e22cfb39b42"). InnerVolumeSpecName "kube-api-access-4sqnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345680    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-kube-api-access-9qn5b" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "kube-api-access-9qn5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345686    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-kube-api-access-dl59j" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "kube-api-access-dl59j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:01:30.345712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345706    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-envoy" pod "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" (UID: "4ad9d225-2e9b-4f14-96e7-2c297cee3ef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.346294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.345756    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-envoy" pod "ad61179b-7e0c-4bde-a7ee-e8272ae300b7" (UID: "ad61179b-7e0c-4bde-a7ee-e8272ae300b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.347064 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.347038    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-envoy" pod "7463fcbe-ed3c-4234-828d-955fa96a394a" (UID: "7463fcbe-ed3c-4234-828d-955fa96a394a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439833    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439863    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439878    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sqnt\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-kube-api-access-4sqnt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439888    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439897    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qn5b\" (UniqueName: \"kubernetes.io/projected/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-kube-api-access-9qn5b\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439906    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl59j\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-kube-api-access-dl59j\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439914    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.439909 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439922    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439930    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439940    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439954    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439961    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439969    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439978    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439986    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.439994    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440003    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shgk5\" (UniqueName: \"kubernetes.io/projected/7463fcbe-ed3c-4234-828d-955fa96a394a-kube-api-access-shgk5\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440011    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440022    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440033    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7463fcbe-ed3c-4234-828d-955fa96a394a-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440041    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440049    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440058    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440066    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440073    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7463fcbe-ed3c-4234-828d-955fa96a394a-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440081    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cae13af-8c9f-4da1-9e48-5e22cfb39b42-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.440313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.440089    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad61179b-7e0c-4bde-a7ee-e8272ae300b7-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:01:30.545332 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.545288    4839 generic.go:356] "Generic (PLEG): container finished" podID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerID="0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d" exitCode=0
Oct 29 23:01:30.545780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.545346    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"
Oct 29 23:01:30.545780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.545372    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" event={"ID":"7463fcbe-ed3c-4234-828d-955fa96a394a","Type":"ContainerDied","Data":"0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d"}
Oct 29 23:01:30.545780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.545420    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv" event={"ID":"7463fcbe-ed3c-4234-828d-955fa96a394a","Type":"ContainerDied","Data":"1adacba6ce777faff85aac9480bc47b06d1e6573d0b49cc2faa1f409d48e6429"}
Oct 29 23:01:30.545780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.545448    4839 scope.go:122] "RemoveContainer" containerID="0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d"
Oct 29 23:01:30.546785 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.546768    4839 generic.go:356] "Generic (PLEG): container finished" podID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerID="0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5" exitCode=0
Oct 29 23:01:30.546908 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.546833    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"
Oct 29 23:01:30.546908 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.546850    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" event={"ID":"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4","Type":"ContainerDied","Data":"0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5"}
Oct 29 23:01:30.546908 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.546906    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg" event={"ID":"4ad9d225-2e9b-4f14-96e7-2c297cee3ef4","Type":"ContainerDied","Data":"185de4c738283f9eb1951951839a09d533f73d1779bcc2201df2b269d56f2607"}
Oct 29 23:01:30.548298 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.548276    4839 generic.go:356] "Generic (PLEG): container finished" podID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerID="59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce" exitCode=0
Oct 29 23:01:30.548441 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.548305    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" event={"ID":"6cae13af-8c9f-4da1-9e48-5e22cfb39b42","Type":"ContainerDied","Data":"59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce"}
Oct 29 23:01:30.548441 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.548327    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"
Oct 29 23:01:30.548441 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.548345    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp" event={"ID":"6cae13af-8c9f-4da1-9e48-5e22cfb39b42","Type":"ContainerDied","Data":"2a15579b21cf5c578984cf57839a667aee02763b5b28a397fe7659cb9e6abfd5"}
Oct 29 23:01:30.550368 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.550349    4839 generic.go:356] "Generic (PLEG): container finished" podID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerID="c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404" exitCode=0
Oct 29 23:01:30.550497 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.550380    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" event={"ID":"ad61179b-7e0c-4bde-a7ee-e8272ae300b7","Type":"ContainerDied","Data":"c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404"}
Oct 29 23:01:30.550497 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.550396    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"
Oct 29 23:01:30.550497 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.550411    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc" event={"ID":"ad61179b-7e0c-4bde-a7ee-e8272ae300b7","Type":"ContainerDied","Data":"1825688121df46d9725fd0a9fec3fa666198811cd595c4b781d188325ab9c150"}
Oct 29 23:01:30.558019 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.557997    4839 scope.go:122] "RemoveContainer" containerID="0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d"
Oct 29 23:01:30.558486 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:30.558315    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d\": container with ID starting with 0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d not found: ID does not exist" containerID="0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d"
Oct 29 23:01:30.558486 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.558348    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d"} err="failed to get container status \"0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d\": rpc error: code = NotFound desc = could not find container \"0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d\": container with ID starting with 0653eb60715d92597e3eebc11175cca5326ec89d04705710bcaf367f9568165d not found: ID does not exist"
Oct 29 23:01:30.558486 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.558372    4839 scope.go:122] "RemoveContainer" containerID="0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5"
Oct 29 23:01:30.569721 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.569703    4839 scope.go:122] "RemoveContainer" containerID="0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5"
Oct 29 23:01:30.570003 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:30.569983    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5\": container with ID starting with 0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5 not found: ID does not exist" containerID="0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5"
Oct 29 23:01:30.570061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.570010    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5"} err="failed to get container status \"0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5\": rpc error: code = NotFound desc = could not find container \"0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5\": container with ID starting with 0b53b5332b84396731b727009b4b3ad693bc0388c8096b65672d289b88daa4d5 not found: ID does not exist"
Oct 29 23:01:30.570061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.570028    4839 scope.go:122] "RemoveContainer" containerID="59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce"
Oct 29 23:01:30.578944 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.578924    4839 scope.go:122] "RemoveContainer" containerID="59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce"
Oct 29 23:01:30.579238 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:30.579218    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce\": container with ID starting with 59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce not found: ID does not exist" containerID="59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce"
Oct 29 23:01:30.579310 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.579246    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce"} err="failed to get container status \"59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce\": rpc error: code = NotFound desc = could not find container \"59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce\": container with ID starting with 59301b24fe123421df40ea68e80cf4d5bad7c56b47d9cdb4d6763aba2e582dce not found: ID does not exist"
Oct 29 23:01:30.579310 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.579263    4839 scope.go:122] "RemoveContainer" containerID="c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404"
Oct 29 23:01:30.588467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.588449    4839 scope.go:122] "RemoveContainer" containerID="c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404"
Oct 29 23:01:30.588770 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:30.588746    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404\": container with ID starting with c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404 not found: ID does not exist" containerID="c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404"
Oct 29 23:01:30.588829 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.588779    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404"} err="failed to get container status \"c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404\": rpc error: code = NotFound desc = could not find container \"c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404\": container with ID starting with c1e107044d67ea1dfded04efa0fbc1c03d102e8b0b319bb83ffb685a5b4fb404 not found: ID does not exist"
Oct 29 23:01:30.606685 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.606657    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"]
Oct 29 23:01:30.609505 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.609475    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-lt9sg"]
Oct 29 23:01:30.645932 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.645892    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"]
Oct 29 23:01:30.649712 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.649683    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-6mfwv"]
Oct 29 23:01:30.720977 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.720879    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"]
Oct 29 23:01:30.724603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.724573    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-zg5zc"]
Oct 29 23:01:30.762127 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.762088    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"]
Oct 29 23:01:30.767176 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:30.767140    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-07eef2-openshift-default-869b7995fc-fsrbp"]
Oct 29 23:01:31.008687 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:31.008610    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f12c21-22f6-4917-b443-2c1caaa6c6b0" path="/var/lib/kubelet/pods/43f12c21-22f6-4917-b443-2c1caaa6c6b0/volumes"
Oct 29 23:01:31.009061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:31.009047    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" path="/var/lib/kubelet/pods/4ad9d225-2e9b-4f14-96e7-2c297cee3ef4/volumes"
Oct 29 23:01:31.009479 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:31.009463    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" path="/var/lib/kubelet/pods/6cae13af-8c9f-4da1-9e48-5e22cfb39b42/volumes"
Oct 29 23:01:31.010194 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:31.010180    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" path="/var/lib/kubelet/pods/7463fcbe-ed3c-4234-828d-955fa96a394a/volumes"
Oct 29 23:01:31.010570 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:31.010559    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" path="/var/lib/kubelet/pods/ad61179b-7e0c-4bde-a7ee-e8272ae300b7/volumes"
Oct 29 23:01:31.608116 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:31.608085    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:33.236949 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:01:33.236908    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:01:33.527900 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:33.527823    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:34.143128 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:34.143092    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:35.263170 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:35.263113    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:36.947662 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:36.947616    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:38.678840 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:38.678807    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:40.288255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:40.288221    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:41.047943 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:41.047906    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"]
Oct 29 23:01:41.061703 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:41.061668    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37dd5804-644qm"]
Oct 29 23:01:41.972371 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:41.972339    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:43.008096 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:43.008061    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30708359-1b51-4b63-b917-ae800aa4ea68" path="/var/lib/kubelet/pods/30708359-1b51-4b63-b917-ae800aa4ea68/volumes"
Oct 29 23:01:43.684936 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:43.684890    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:45.601013 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:45.599376    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:47.516467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:47.516429    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:49.409880 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:49.409844    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:51.255917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:51.255875    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:53.200027 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:53.199991    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:55.108032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:55.107990    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:56.857205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:56.857173    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:58.035191 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:58.035143    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"]
Oct 29 23:01:58.049838 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:58.049803    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d48892b-tz4wk"]
Oct 29 23:01:58.893883 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:58.893849    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:01:59.007052 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:01:59.007018    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa249b06-17bc-4da1-8bd2-3061084bb1cd" path="/var/lib/kubelet/pods/aa249b06-17bc-4da1-8bd2-3061084bb1cd/volumes"
Oct 29 23:02:00.579309 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:00.579275    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:02.135692 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:02.135655    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:03.237586 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:03.237542    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:02:03.759068 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:03.759032    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:05.505374 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:05.505337    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:07.317291 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:07.317252    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:08.987804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:08.987767    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:10.769847 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:10.769810    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:12.413861 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:12.413831    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:14.079441 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:14.079407    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:14.183766 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:14.183729    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:15.028622 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:15.028588    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"]
Oct 29 23:02:15.040978 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:15.040950    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9977110e-rffhp"]
Oct 29 23:02:15.920769 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:15.920736    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:16.044123 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:16.044088    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:17.007460 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:17.007427    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219fa173-07df-42dc-9cb8-c5f5d2043fb0" path="/var/lib/kubelet/pods/219fa173-07df-42dc-9cb8-c5f5d2043fb0/volumes"
Oct 29 23:02:17.677594 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:17.677559    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:17.778699 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:17.778664    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:19.243234 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:19.243178    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:19.621198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:19.621161    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:21.033557 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:21.033520    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"]
Oct 29 23:02:21.046784 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:21.046756    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc1b5fe5-5vnx8"]
Oct 29 23:02:21.163770 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:21.163741    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:22.886660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:22.886623    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:23.008991 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:23.008951    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983666a3-7eba-422d-b110-a8ad96351dc1" path="/var/lib/kubelet/pods/983666a3-7eba-422d-b110-a8ad96351dc1/volumes"
Oct 29 23:02:24.514140 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:24.514103    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:24.759251 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:24.759215    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"]
Oct 29 23:02:24.796309 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:24.796278    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"]
Oct 29 23:02:24.901642 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:24.901599    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:25.005780 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:25.005744    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:25.048117 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:25.048034    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"]
Oct 29 23:02:25.048117 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:25.048086    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574b2b00-gww2q"]
Oct 29 23:02:27.008161 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:27.008121    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff320260-9e76-4ee5-b8aa-9910bd6e3c25" path="/var/lib/kubelet/pods/ff320260-9e76-4ee5-b8aa-9910bd6e3c25/volumes"
Oct 29 23:02:29.760336 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:29.760286    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" podUID="adab7618-52c4-4106-a1f2-76d19ab11656" containerName="payload-processing" containerID="cri-o://99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807" gracePeriod=25
Oct 29 23:02:29.796927 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:29.796883    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" podUID="9904b506-4dc7-44a9-a5a3-bb413392aa03" containerName="payload-pre-processing" containerID="cri-o://8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7" gracePeriod=25
Oct 29 23:02:33.238777 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:33.238731    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:02:37.548256 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:37.548173    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" podUID="adab7618-52c4-4106-a1f2-76d19ab11656" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:37.548256 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:37.548183    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" podUID="9904b506-4dc7-44a9-a5a3-bb413392aa03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:41.069205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.069168    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"]
Oct 29 23:02:41.069798 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.069431    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp" podUID="eed11e89-0ac8-4cbf-b997-1a35e785af7f" containerName="maas-api" containerID="cri-o://22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08" gracePeriod=30
Oct 29 23:02:41.152691 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.152656    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:41.408210 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.407739    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 23:02:41.446267 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.446235    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed11e89-0ac8-4cbf-b997-1a35e785af7f-kube-api-access-jzrpj\" (UniqueName: \"kubernetes.io/projected/eed11e89-0ac8-4cbf-b997-1a35e785af7f-kube-api-access-jzrpj\") pod \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") "
Oct 29 23:02:41.446477 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.446298    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-maas-api-tls\") pod \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") "
Oct 29 23:02:41.446477 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.446331    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed11e89-0ac8-4cbf-b997-1a35e785af7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed11e89-0ac8-4cbf-b997-1a35e785af7f-openshift-service-ca\") pod \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") "
Oct 29 23:02:41.446477 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.446389    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-metrics-certs\") pod \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\" (UID: \"eed11e89-0ac8-4cbf-b997-1a35e785af7f\") "
Oct 29 23:02:41.447019 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.446987    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed11e89-0ac8-4cbf-b997-1a35e785af7f-openshift-service-ca" pod "eed11e89-0ac8-4cbf-b997-1a35e785af7f" (UID: "eed11e89-0ac8-4cbf-b997-1a35e785af7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:41.448936 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.448913    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-metrics-certs" pod "eed11e89-0ac8-4cbf-b997-1a35e785af7f" (UID: "eed11e89-0ac8-4cbf-b997-1a35e785af7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:41.449022 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.448940    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-maas-api-tls" pod "eed11e89-0ac8-4cbf-b997-1a35e785af7f" (UID: "eed11e89-0ac8-4cbf-b997-1a35e785af7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:02:41.449022 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.448952    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed11e89-0ac8-4cbf-b997-1a35e785af7f-kube-api-access-jzrpj" pod "eed11e89-0ac8-4cbf-b997-1a35e785af7f" (UID: "eed11e89-0ac8-4cbf-b997-1a35e785af7f"). InnerVolumeSpecName "kube-api-access-jzrpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:41.552163 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.550825    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-maas-api-tls\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:41.552163 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.550866    4839 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed11e89-0ac8-4cbf-b997-1a35e785af7f-openshift-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:41.552163 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.550882    4839 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed11e89-0ac8-4cbf-b997-1a35e785af7f-metrics-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:41.552163 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.550914    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzrpj\" (UniqueName: \"kubernetes.io/projected/eed11e89-0ac8-4cbf-b997-1a35e785af7f-kube-api-access-jzrpj\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:41.683243 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.683130    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:41.783117 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:41.783082    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:42.029358 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.029318    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"]
Oct 29 23:02:42.032382 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.032352    4839 generic.go:356] "Generic (PLEG): container finished" podID="eed11e89-0ac8-4cbf-b997-1a35e785af7f" containerID="22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08" exitCode=0
Oct 29 23:02:42.032535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.032399    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"
Oct 29 23:02:42.032535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.032447    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp" event={"ID":"eed11e89-0ac8-4cbf-b997-1a35e785af7f","Type":"ContainerDied","Data":"22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08"}
Oct 29 23:02:42.032535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.032496    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp" event={"ID":"eed11e89-0ac8-4cbf-b997-1a35e785af7f","Type":"ContainerDied","Data":"9949d1aa2f0e0533781d5d66061e20d326d6e3691f1299cb655e2c668d1f2c14"}
Oct 29 23:02:42.032535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.032520    4839 scope.go:122] "RemoveContainer" containerID="22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08"
Oct 29 23:02:42.043621 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.043589    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b780ad0-nzfv6"]
Oct 29 23:02:42.043693 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.043677    4839 scope.go:122] "RemoveContainer" containerID="22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08"
Oct 29 23:02:42.043986 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:42.043966    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08\": container with ID starting with 22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08 not found: ID does not exist" containerID="22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08"
Oct 29 23:02:42.044056 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.043992    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08"} err="failed to get container status \"22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08\": rpc error: code = NotFound desc = could not find container \"22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08\": container with ID starting with 22e60005747ba3418922e77b2862f9c3442b5554d97047687d4b778aa5173a08 not found: ID does not exist"
Oct 29 23:02:42.089857 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.089824    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"]
Oct 29 23:02:42.093076 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.093053    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-ffba57-5b58bf5769-swtfp"]
Oct 29 23:02:42.341537 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:42.341456    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:43.009030 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:43.008996    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205" path="/var/lib/kubelet/pods/2da4be8e-44a6-4ab1-a36c-6f4a2d1e7205/volumes"
Oct 29 23:02:43.009455 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:43.009441    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed11e89-0ac8-4cbf-b997-1a35e785af7f" path="/var/lib/kubelet/pods/eed11e89-0ac8-4cbf-b997-1a35e785af7f/volumes"
Oct 29 23:02:47.548476 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:47.548436    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" podUID="adab7618-52c4-4106-a1f2-76d19ab11656" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:47.548901 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:47.548444    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" podUID="9904b506-4dc7-44a9-a5a3-bb413392aa03" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:02:49.684117 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:49.684083    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:49.807379 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:49.807347    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:50.040520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.040485    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"]
Oct 29 23:02:50.041799 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.041765    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy" containerID="cri-o://fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574" gracePeriod=30
Oct 29 23:02:50.054733 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.054698    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"]
Oct 29 23:02:50.054922 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.054898    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy" containerID="cri-o://3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc" gracePeriod=30
Oct 29 23:02:50.070022 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.069128    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"]
Oct 29 23:02:50.070022 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.069421    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" containerID="cri-o://9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a" gracePeriod=30
Oct 29 23:02:50.077195 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.077130    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"]
Oct 29 23:02:50.077442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.077400    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy" containerID="cri-o://8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d" gracePeriod=30
Oct 29 23:02:50.077573 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.077469    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 23:02:50.089443 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.089418    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 23:02:50.092822 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.092794    4839 generic.go:356] "Generic (PLEG): container finished" podID="adab7618-52c4-4106-a1f2-76d19ab11656" containerID="99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807" exitCode=0
Oct 29 23:02:50.092929 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.092832    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"
Oct 29 23:02:50.092929 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.092865    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" event={"ID":"adab7618-52c4-4106-a1f2-76d19ab11656","Type":"ContainerDied","Data":"99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807"}
Oct 29 23:02:50.092929 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.092917    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk" event={"ID":"adab7618-52c4-4106-a1f2-76d19ab11656","Type":"ContainerDied","Data":"b66d63da7e4fb3871e9280e23708fc5fa6a1752a8373487c0c7bd68082633dca"}
Oct 29 23:02:50.093095 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.092946    4839 scope.go:122] "RemoveContainer" containerID="99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807"
Oct 29 23:02:50.094360 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.094336    4839 generic.go:356] "Generic (PLEG): container finished" podID="9904b506-4dc7-44a9-a5a3-bb413392aa03" containerID="8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7" exitCode=0
Oct 29 23:02:50.094445 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.094380    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"
Oct 29 23:02:50.094445 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.094395    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" event={"ID":"9904b506-4dc7-44a9-a5a3-bb413392aa03","Type":"ContainerDied","Data":"8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7"}
Oct 29 23:02:50.094445 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.094433    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649" event={"ID":"9904b506-4dc7-44a9-a5a3-bb413392aa03","Type":"ContainerDied","Data":"43b43a14beae010e4acf2dd04e455d0a60f417cde26ee71f6bc2a3432b02e445"}
Oct 29 23:02:50.102243 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.102223    4839 scope.go:122] "RemoveContainer" containerID="99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807"
Oct 29 23:02:50.102533 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:50.102506    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807\": container with ID starting with 99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807 not found: ID does not exist" containerID="99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807"
Oct 29 23:02:50.102636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.102540    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807"} err="failed to get container status \"99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807\": rpc error: code = NotFound desc = could not find container \"99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807\": container with ID starting with 99afdda3c626ab63304a41908a4a688d044db3790ba1ccfd668b8b1818e84807 not found: ID does not exist"
Oct 29 23:02:50.102636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.102564    4839 scope.go:122] "RemoveContainer" containerID="8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7"
Oct 29 23:02:50.111570 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.111538    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:50.112890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.112874    4839 scope.go:122] "RemoveContainer" containerID="8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7"
Oct 29 23:02:50.113212 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:50.113191    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7\": container with ID starting with 8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7 not found: ID does not exist" containerID="8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7"
Oct 29 23:02:50.113306 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.113218    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7"} err="failed to get container status \"8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7\": rpc error: code = NotFound desc = could not find container \"8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7\": container with ID starting with 8a9415ae6981509e1bc80114c66689e3850ebd4e421c6e4ebc988f6e523305d7 not found: ID does not exist"
Oct 29 23:02:50.151709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.146745    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adab7618-52c4-4106-a1f2-76d19ab11656-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adab7618-52c4-4106-a1f2-76d19ab11656-plugins-config-volume\") pod \"adab7618-52c4-4106-a1f2-76d19ab11656\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") "
Oct 29 23:02:50.151709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.146854    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9904b506-4dc7-44a9-a5a3-bb413392aa03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9904b506-4dc7-44a9-a5a3-bb413392aa03-plugins-config-volume\") pod \"9904b506-4dc7-44a9-a5a3-bb413392aa03\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") "
Oct 29 23:02:50.151709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.146901    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9904b506-4dc7-44a9-a5a3-bb413392aa03-kube-api-access-b8457\" (UniqueName: \"kubernetes.io/projected/9904b506-4dc7-44a9-a5a3-bb413392aa03-kube-api-access-b8457\") pod \"9904b506-4dc7-44a9-a5a3-bb413392aa03\" (UID: \"9904b506-4dc7-44a9-a5a3-bb413392aa03\") "
Oct 29 23:02:50.151709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.146949    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adab7618-52c4-4106-a1f2-76d19ab11656-kube-api-access-w67b9\" (UniqueName: \"kubernetes.io/projected/adab7618-52c4-4106-a1f2-76d19ab11656-kube-api-access-w67b9\") pod \"adab7618-52c4-4106-a1f2-76d19ab11656\" (UID: \"adab7618-52c4-4106-a1f2-76d19ab11656\") "
Oct 29 23:02:50.151709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.149472    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9904b506-4dc7-44a9-a5a3-bb413392aa03-plugins-config-volume" pod "9904b506-4dc7-44a9-a5a3-bb413392aa03" (UID: "9904b506-4dc7-44a9-a5a3-bb413392aa03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:50.151709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.151624    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adab7618-52c4-4106-a1f2-76d19ab11656-plugins-config-volume" pod "adab7618-52c4-4106-a1f2-76d19ab11656" (UID: "adab7618-52c4-4106-a1f2-76d19ab11656"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:50.152198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.152082    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adab7618-52c4-4106-a1f2-76d19ab11656-kube-api-access-w67b9" pod "adab7618-52c4-4106-a1f2-76d19ab11656" (UID: "adab7618-52c4-4106-a1f2-76d19ab11656"). InnerVolumeSpecName "kube-api-access-w67b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:50.159088 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.159055    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9904b506-4dc7-44a9-a5a3-bb413392aa03-kube-api-access-b8457" pod "9904b506-4dc7-44a9-a5a3-bb413392aa03" (UID: "9904b506-4dc7-44a9-a5a3-bb413392aa03"). InnerVolumeSpecName "kube-api-access-b8457". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:50.248722 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.248620    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adab7618-52c4-4106-a1f2-76d19ab11656-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:50.248722 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.248647    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9904b506-4dc7-44a9-a5a3-bb413392aa03-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:50.248722 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.248660    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8457\" (UniqueName: \"kubernetes.io/projected/9904b506-4dc7-44a9-a5a3-bb413392aa03-kube-api-access-b8457\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:50.248722 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.248670    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w67b9\" (UniqueName: \"kubernetes.io/projected/adab7618-52c4-4106-a1f2-76d19ab11656-kube-api-access-w67b9\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:50.444047 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.444011    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"]
Oct 29 23:02:50.449000 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.448968    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-ffba57-746df89455-5fsrk"]
Oct 29 23:02:50.489977 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.489941    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"]
Oct 29 23:02:50.492727 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.492702    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ffba57-7f588658d7-2m649"]
Oct 29 23:02:50.814062 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.814022    4839 patch_prober.go:29] interesting pod/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 29 23:02:50.814469 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:50.814092    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 29 23:02:51.007448 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:51.007414    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9904b506-4dc7-44a9-a5a3-bb413392aa03" path="/var/lib/kubelet/pods/9904b506-4dc7-44a9-a5a3-bb413392aa03/volumes"
Oct 29 23:02:51.007789 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:51.007777    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adab7618-52c4-4106-a1f2-76d19ab11656" path="/var/lib/kubelet/pods/adab7618-52c4-4106-a1f2-76d19ab11656/volumes"
Oct 29 23:02:51.031658 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:51.031574    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"]
Oct 29 23:02:51.033717 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:51.033693    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b4209b7-8np2h"]
Oct 29 23:02:53.009206 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:53.009172    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8765bc57-d592-4213-a88c-e5462620b637" path="/var/lib/kubelet/pods/8765bc57-d592-4213-a88c-e5462620b637/volumes"
Oct 29 23:02:55.355332 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.355302    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 23:02:55.391510 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.391262    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399312    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-istio-token\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-istio-token\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399388    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-kube-api-access-xzc29\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-kube-api-access-xzc29\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399428    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0486bcec-d668-4e37-a171-d1ef285293e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0486bcec-d668-4e37-a171-d1ef285293e9-istiod-ca-cert\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399459    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-data\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399543    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-certs\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399604    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0486bcec-d668-4e37-a171-d1ef285293e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0486bcec-d668-4e37-a171-d1ef285293e9-istio-podinfo\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399638    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-socket\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399676    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-credential-socket\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.399747    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-envoy\") pod \"0486bcec-d668-4e37-a171-d1ef285293e9\" (UID: \"0486bcec-d668-4e37-a171-d1ef285293e9\") "
Oct 29 23:02:55.404808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.402560    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-envoy" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.405457 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.405429    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-certs" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.406513 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.405907    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0486bcec-d668-4e37-a171-d1ef285293e9-istiod-ca-cert" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.406513 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.406032    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-istio-token" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.406513 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.406179    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-data" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.406513 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.406365    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-socket" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.406797 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.406567    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-credential-socket" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.407849 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.407818    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-kube-api-access-xzc29" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "kube-api-access-xzc29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.408735 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.408526    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0486bcec-d668-4e37-a171-d1ef285293e9-istio-podinfo" pod "0486bcec-d668-4e37-a171-d1ef285293e9" (UID: "0486bcec-d668-4e37-a171-d1ef285293e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:55.412714 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.412689    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 23:02:55.417613 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.417589    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 23:02:55.501179 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501103    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-kube-api-access-kkrqm\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-kube-api-access-kkrqm\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.501179 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501169    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-podinfo\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.501440 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501201    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-kube-api-access-lrph2\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-kube-api-access-lrph2\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.501440 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501231    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-certs\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.501440 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501263    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-kube-api-access-49gvq\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-kube-api-access-49gvq\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.501440 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501305    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-credential-socket\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.501440 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501375    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-podinfo\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.501440 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501414    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d687941a-088e-4354-9d55-98861c7f308a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687941a-088e-4354-9d55-98861c7f308a-istiod-ca-cert\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501444    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-credential-socket\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501498    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-token\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-token\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501555    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-envoy\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501583    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-envoy\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501628    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-data\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501656    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d687941a-088e-4354-9d55-98861c7f308a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687941a-088e-4354-9d55-98861c7f308a-istio-podinfo\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501684    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-token\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-token\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.501758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501728    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-certs\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501772    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-envoy\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501799    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-socket\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501829    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istiod-ca-cert\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501858    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-data\") pod \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\" (UID: \"89b4bcad-310b-4c0a-a691-d5319b2f01bd\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501906    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-credential-socket\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.501972    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-certs\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502005    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30d17cf-246c-4524-a1b2-78d4c34fc793-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30d17cf-246c-4524-a1b2-78d4c34fc793-istiod-ca-cert\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502033    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-socket\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502063    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-data\") pod \"b30d17cf-246c-4524-a1b2-78d4c34fc793\" (UID: \"b30d17cf-246c-4524-a1b2-78d4c34fc793\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502123    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-socket\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.502183 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502165    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-istio-token\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-istio-token\") pod \"d687941a-088e-4354-9d55-98861c7f308a\" (UID: \"d687941a-088e-4354-9d55-98861c7f308a\") "
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502557    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0486bcec-d668-4e37-a171-d1ef285293e9-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502572    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502587    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502601    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502614    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502627    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzc29\" (UniqueName: \"kubernetes.io/projected/0486bcec-d668-4e37-a171-d1ef285293e9-kube-api-access-xzc29\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502641    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0486bcec-d668-4e37-a171-d1ef285293e9-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502654    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.502667    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0486bcec-d668-4e37-a171-d1ef285293e9-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.503180    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-data" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.503579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.503482    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-credential-socket" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.504237 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.503679    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-certs" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.504237 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.503966    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30d17cf-246c-4524-a1b2-78d4c34fc793-istiod-ca-cert" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.504237 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.504214    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-socket" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.504398 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.504247    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-data" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.504398 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.504386    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-data" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.504574 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.504555    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-socket" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.505008 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.504850    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-certs" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.505422 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.505240    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-socket" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.505867 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.505635    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d687941a-088e-4354-9d55-98861c7f308a-istiod-ca-cert" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.507444 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.507415    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-kube-api-access-kkrqm" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "kube-api-access-kkrqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.507707 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.507640    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-credential-socket" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.507822 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.507806    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-certs" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.508889 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.508628    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-envoy" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.510049 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.509813    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-credential-socket" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.511188 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.511043    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-kube-api-access-49gvq" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "kube-api-access-49gvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.511390 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.511371    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istiod-ca-cert" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:02:55.511917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.511896    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-kube-api-access-lrph2" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "kube-api-access-lrph2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.514071 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.514047    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-podinfo" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:55.516684 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.516652    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-istio-token" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.516815 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.516800    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-envoy" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.517006 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.516979    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-podinfo" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:55.517374 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.517320    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-token" pod "b30d17cf-246c-4524-a1b2-78d4c34fc793" (UID: "b30d17cf-246c-4524-a1b2-78d4c34fc793"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.517567 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.517542    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-token" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:02:55.518425 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.518400    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d687941a-088e-4354-9d55-98861c7f308a-istio-podinfo" pod "d687941a-088e-4354-9d55-98861c7f308a" (UID: "d687941a-088e-4354-9d55-98861c7f308a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 29 23:02:55.519877 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.519844    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-envoy" pod "89b4bcad-310b-4c0a-a691-d5319b2f01bd" (UID: "89b4bcad-310b-4c0a-a691-d5319b2f01bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604020    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604065    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d687941a-088e-4354-9d55-98861c7f308a-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604083    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604099    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604113    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604126    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604139    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604171    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d687941a-088e-4354-9d55-98861c7f308a-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604184    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604197    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604210    4839 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-envoy\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604225    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604238    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604252    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604265    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604279    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604293    4839 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30d17cf-246c-4524-a1b2-78d4c34fc793-istiod-ca-cert\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604306    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604320    4839 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-istio-data\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.604325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604334    4839 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.605667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604347    4839 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-istio-token\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.605667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604361    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrqm\" (UniqueName: \"kubernetes.io/projected/d687941a-088e-4354-9d55-98861c7f308a-kube-api-access-kkrqm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.605667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604376    4839 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89b4bcad-310b-4c0a-a691-d5319b2f01bd-istio-podinfo\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.605667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604391    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrph2\" (UniqueName: \"kubernetes.io/projected/89b4bcad-310b-4c0a-a691-d5319b2f01bd-kube-api-access-lrph2\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.605667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604406    4839 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d687941a-088e-4354-9d55-98861c7f308a-workload-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.605667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604420    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gvq\" (UniqueName: \"kubernetes.io/projected/b30d17cf-246c-4524-a1b2-78d4c34fc793-kube-api-access-49gvq\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.605667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.604433    4839 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30d17cf-246c-4524-a1b2-78d4c34fc793-credential-socket\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:02:55.960676 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:55.958278    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:56.095972 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.095466    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:56.141832 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.141796    4839 generic.go:356] "Generic (PLEG): container finished" podID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerID="9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a" exitCode=0
Oct 29 23:02:56.143680 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.141859    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"
Oct 29 23:02:56.144292 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.141879    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" event={"ID":"b30d17cf-246c-4524-a1b2-78d4c34fc793","Type":"ContainerDied","Data":"9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a"}
Oct 29 23:02:56.144417 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.144355    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p" event={"ID":"b30d17cf-246c-4524-a1b2-78d4c34fc793","Type":"ContainerDied","Data":"1d606112affcfb30bb48052e824bb00969dcb31eb45e9b58499e47686d0c02d5"}
Oct 29 23:02:56.144417 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.144397    4839 scope.go:122] "RemoveContainer" containerID="9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a"
Oct 29 23:02:56.146505 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.146480    4839 generic.go:356] "Generic (PLEG): container finished" podID="0486bcec-d668-4e37-a171-d1ef285293e9" containerID="fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574" exitCode=0
Oct 29 23:02:56.146689 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.146526    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"
Oct 29 23:02:56.146793 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.146546    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" event={"ID":"0486bcec-d668-4e37-a171-d1ef285293e9","Type":"ContainerDied","Data":"fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574"}
Oct 29 23:02:56.146872 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.146814    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs" event={"ID":"0486bcec-d668-4e37-a171-d1ef285293e9","Type":"ContainerDied","Data":"e268be4c4468e3fc91e4a3558d859ea102521151a62bf3eda7e5ce7ac6204745"}
Oct 29 23:02:56.151849 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.151821    4839 generic.go:356] "Generic (PLEG): container finished" podID="d687941a-088e-4354-9d55-98861c7f308a" containerID="3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc" exitCode=0
Oct 29 23:02:56.151970 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.151903    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" event={"ID":"d687941a-088e-4354-9d55-98861c7f308a","Type":"ContainerDied","Data":"3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc"}
Oct 29 23:02:56.151970 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.151957    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp" event={"ID":"d687941a-088e-4354-9d55-98861c7f308a","Type":"ContainerDied","Data":"0af8eb18721c48b871b49af7b634194381287497d2878f433c01a5a1d39926af"}
Oct 29 23:02:56.152569 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.152482    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"
Oct 29 23:02:56.153838 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.153815    4839 generic.go:356] "Generic (PLEG): container finished" podID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerID="8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d" exitCode=0
Oct 29 23:02:56.153938 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.153859    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" event={"ID":"89b4bcad-310b-4c0a-a691-d5319b2f01bd","Type":"ContainerDied","Data":"8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d"}
Oct 29 23:02:56.153938 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.153897    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf" event={"ID":"89b4bcad-310b-4c0a-a691-d5319b2f01bd","Type":"ContainerDied","Data":"6e0c2d46af8c7ddffa2a59df360d8b5130da68ba9aaf0eec0077ec2db8e7f065"}
Oct 29 23:02:56.153938 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.153862    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"
Oct 29 23:02:56.163854 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.163813    4839 scope.go:122] "RemoveContainer" containerID="9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a"
Oct 29 23:02:56.164233 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:56.164205    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a\": container with ID starting with 9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a not found: ID does not exist" containerID="9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a"
Oct 29 23:02:56.164349 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.164247    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a"} err="failed to get container status \"9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a\": rpc error: code = NotFound desc = could not find container \"9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a\": container with ID starting with 9e04821bcadcbd2a17c6dfe9088d0ab8b7c2a30547268ae2ebd0f0bf96c3287a not found: ID does not exist"
Oct 29 23:02:56.164349 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.164269    4839 scope.go:122] "RemoveContainer" containerID="fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574"
Oct 29 23:02:56.200555 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.200444    4839 scope.go:122] "RemoveContainer" containerID="fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574"
Oct 29 23:02:56.200866 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:56.200832    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574\": container with ID starting with fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574 not found: ID does not exist" containerID="fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574"
Oct 29 23:02:56.200956 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.200877    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574"} err="failed to get container status \"fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574\": rpc error: code = NotFound desc = could not find container \"fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574\": container with ID starting with fc4a9170e14bd75300bb666e5f6653a0e531d2ffeb54d62b711ec84ec44f3574 not found: ID does not exist"
Oct 29 23:02:56.200956 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.200903    4839 scope.go:122] "RemoveContainer" containerID="3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc"
Oct 29 23:02:56.215345 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.215321    4839 scope.go:122] "RemoveContainer" containerID="3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc"
Oct 29 23:02:56.215791 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:56.215742    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc\": container with ID starting with 3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc not found: ID does not exist" containerID="3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc"
Oct 29 23:02:56.216282 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.215828    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc"} err="failed to get container status \"3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc\": rpc error: code = NotFound desc = could not find container \"3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc\": container with ID starting with 3ad1011de6a885169e28953b0c5e74fe80312123fbad58e2bf8e391107010adc not found: ID does not exist"
Oct 29 23:02:56.216282 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.215889    4839 scope.go:122] "RemoveContainer" containerID="8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d"
Oct 29 23:02:56.229319 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.229295    4839 scope.go:122] "RemoveContainer" containerID="8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d"
Oct 29 23:02:56.229789 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:02:56.229762    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d\": container with ID starting with 8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d not found: ID does not exist" containerID="8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d"
Oct 29 23:02:56.229885 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.229802    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d"} err="failed to get container status \"8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d\": rpc error: code = NotFound desc = could not find container \"8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d\": container with ID starting with 8870bb40a6fc05a22cc6661f9cc5c55f00abed3442a26bd7d80df8c6bb576b0d not found: ID does not exist"
Oct 29 23:02:56.262320 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.262279    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"]
Oct 29 23:02:56.275136 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.275097    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-bkx6p"]
Oct 29 23:02:56.305640 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.305604    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"]
Oct 29 23:02:56.310770 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.310737    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-x49sf"]
Oct 29 23:02:56.348055 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.348016    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"]
Oct 29 23:02:56.351782 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.351751    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-mpghs"]
Oct 29 23:02:56.394309 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.394220    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"]
Oct 29 23:02:56.399739 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:56.399707    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ffba57-openshift-default-9778845f9-s87wp"]
Oct 29 23:02:57.034963 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:57.034925    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" path="/var/lib/kubelet/pods/0486bcec-d668-4e37-a171-d1ef285293e9/volumes"
Oct 29 23:02:57.035840 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:57.035812    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" path="/var/lib/kubelet/pods/89b4bcad-310b-4c0a-a691-d5319b2f01bd/volumes"
Oct 29 23:02:57.036553 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:57.036531    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" path="/var/lib/kubelet/pods/b30d17cf-246c-4524-a1b2-78d4c34fc793/volumes"
Oct 29 23:02:57.041659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:57.041633    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d687941a-088e-4354-9d55-98861c7f308a" path="/var/lib/kubelet/pods/d687941a-088e-4354-9d55-98861c7f308a/volumes"
Oct 29 23:02:57.640171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:57.639608    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:02:59.029166 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:59.029110    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"]
Oct 29 23:02:59.042384 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:02:59.042353    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0c6ee35-xw59j"]
Oct 29 23:03:01.007306 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:01.007275    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198ed509-6ae3-4382-b9b1-16065f113c0c" path="/var/lib/kubelet/pods/198ed509-6ae3-4382-b9b1-16065f113c0c/volumes"
Oct 29 23:03:03.237566 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:03:03.237529    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:03:03.584990 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:03.584947    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:12.025886 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:12.025843    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"]
Oct 29 23:03:12.039013 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:12.038985    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d91cde7-ll9zq"]
Oct 29 23:03:13.007241 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:13.007200    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114aa2b3-f96b-4429-ad1b-27c9e4442095" path="/var/lib/kubelet/pods/114aa2b3-f96b-4429-ad1b-27c9e4442095/volumes"
Oct 29 23:03:15.987666 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:15.987632    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:16.441232 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441195    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"]
Oct 29 23:03:16.441579 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441565    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441581    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441594    4839 cpu_manager.go:414] "containerMap: removing container" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441599    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441613    4839 cpu_manager.go:414] "containerMap: removing container" podUID="adab7618-52c4-4106-a1f2-76d19ab11656" containerName="payload-processing"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441622    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adab7618-52c4-4106-a1f2-76d19ab11656" containerName="payload-processing"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441629    4839 cpu_manager.go:414] "containerMap: removing container" podUID="9904b506-4dc7-44a9-a5a3-bb413392aa03" containerName="payload-pre-processing"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441633    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9904b506-4dc7-44a9-a5a3-bb413392aa03" containerName="payload-pre-processing"
Oct 29 23:03:16.441637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441639    4839 cpu_manager.go:414] "containerMap: removing container" podUID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" containerName="payload-pre-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441644    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" containerName="payload-pre-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441651    4839 cpu_manager.go:414] "containerMap: removing container" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441656    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441663    4839 cpu_manager.go:414] "containerMap: removing container" podUID="5943fc94-a232-47b8-9efb-01b669ed2787" containerName="payload-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441668    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5943fc94-a232-47b8-9efb-01b669ed2787" containerName="payload-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441677    4839 cpu_manager.go:414] "containerMap: removing container" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441682    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441694    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" containerName="revoke-keys"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441700    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" containerName="revoke-keys"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441706    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441710    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441717    4839 cpu_manager.go:414] "containerMap: removing container" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441721    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441727    4839 cpu_manager.go:414] "containerMap: removing container" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441734    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441741    4839 cpu_manager.go:414] "containerMap: removing container" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441746    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441751    4839 cpu_manager.go:414] "containerMap: removing container" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441756    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441763    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441770    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441781    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a8454cc6-c9b0-4a21-a803-7088233ace04" containerName="payload-pre-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441785    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8454cc6-c9b0-4a21-a803-7088233ace04" containerName="payload-pre-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441795    4839 cpu_manager.go:414] "containerMap: removing container" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441800    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441807    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441811    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441817    4839 cpu_manager.go:414] "containerMap: removing container" podUID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" containerName="payload-pre-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441822    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" containerName="payload-pre-processing"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441827    4839 cpu_manager.go:414] "containerMap: removing container" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441832    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441838    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441845    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441855    4839 cpu_manager.go:414] "containerMap: removing container" podUID="eed11e89-0ac8-4cbf-b997-1a35e785af7f" containerName="maas-api"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441860    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed11e89-0ac8-4cbf-b997-1a35e785af7f" containerName="maas-api"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441865    4839 cpu_manager.go:414] "containerMap: removing container" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441870    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441875    4839 cpu_manager.go:414] "containerMap: removing container" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441879    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441886    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441890    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441897    4839 cpu_manager.go:414] "containerMap: removing container" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy"
Oct 29 23:03:16.441881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441902    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441909    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" containerName="payload-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441915    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" containerName="payload-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441950    4839 cpu_manager.go:414] "containerMap: removing container" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441956    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441962    4839 cpu_manager.go:414] "containerMap: removing container" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441966    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441972    4839 cpu_manager.go:414] "containerMap: removing container" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441976    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441981    4839 cpu_manager.go:414] "containerMap: removing container" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441986    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.441995    4839 cpu_manager.go:414] "containerMap: removing container" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442001    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442009    4839 cpu_manager.go:414] "containerMap: removing container" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442014    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442019    4839 cpu_manager.go:414] "containerMap: removing container" podUID="34e35069-c61b-4d56-b47e-70b08f54dffe" containerName="revoke-keys"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442022    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e35069-c61b-4d56-b47e-70b08f54dffe" containerName="revoke-keys"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442084    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="adab7618-52c4-4106-a1f2-76d19ab11656" containerName="payload-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442091    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="9904b506-4dc7-44a9-a5a3-bb413392aa03" containerName="payload-pre-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442098    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b24ac3e-d12b-41fb-b648-a97fb3ccdc3d" containerName="payload-pre-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442103    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="74110f0b-b412-42f2-884e-0438f0fc02a1" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442109    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7929816-c979-45d3-8dda-4688e78bc292" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442115    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="5943fc94-a232-47b8-9efb-01b669ed2787" containerName="payload-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442120    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3f0c9cf-92b3-4b5f-9818-1643ce63672f" containerName="main"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442126    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" containerName="revoke-keys"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442131    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad61179b-7e0c-4bde-a7ee-e8272ae300b7" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442136    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="0486bcec-d668-4e37-a171-d1ef285293e9" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442141    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442160    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd1ec599-e3bc-40f4-bc65-ab513668d68b" containerName="main"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442168    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30d17cf-246c-4524-a1b2-78d4c34fc793" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442176    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8454cc6-c9b0-4a21-a803-7088233ace04" containerName="payload-pre-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442183    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442188    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d687941a-088e-4354-9d55-98861c7f308a" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442194    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="db34ef8b-313c-4614-b15d-09a6ac3dbe63" containerName="payload-pre-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442200    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="1472f1e3-2edc-4ce5-9d92-4603e6bfc87f" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442205    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28d4a2f-4950-471a-8de9-bba72662c7b8" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442209    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442214    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed11e89-0ac8-4cbf-b997-1a35e785af7f" containerName="maas-api"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442219    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="68408cd1-eda7-4df5-80ad-e82f9cf809e9" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442223    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cae13af-8c9f-4da1-9e48-5e22cfb39b42" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442228    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2821795-ad5a-4f7c-b0ea-080a8437f717" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442233    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b4bcad-310b-4c0a-a691-d5319b2f01bd" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442240    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48caf36-726a-4a6a-8dbb-990fb2c03bff" containerName="payload-processing"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442248    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f5a414-18f6-4063-9e8a-62d7cc7913dd" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442255    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="7463fcbe-ed3c-4234-828d-955fa96a394a" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442259    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f4df42-81b7-4660-82b4-04c567edc874" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442265    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="5397ab93-f411-494f-a840-5bfea3e38c39" containerName="main"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442269    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad9d225-2e9b-4f14-96e7-2c297cee3ef4" containerName="istio-proxy"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442272    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e35069-c61b-4d56-b47e-70b08f54dffe" containerName="revoke-keys"
Oct 29 23:03:16.443255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.442277    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfda4363-0693-4a7e-8ea5-8babe3a0bfe3" containerName="main"
Oct 29 23:03:16.445558 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.445542    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.463589 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.463559    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"]
Oct 29 23:03:16.465926 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.465898    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:03:16.474364 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.474340    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:03:16.474695 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.474665    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:03:16.479501 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.479483    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:03:16.543629 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.543590    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.543778 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.543711    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.543778 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.543744    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2tr\" (UniqueName: \"kubernetes.io/projected/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-kube-api-access-8q2tr\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.644914 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.644872    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2tr\" (UniqueName: \"kubernetes.io/projected/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-kube-api-access-8q2tr\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.645092 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.644925    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.645092 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.644988    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.645721 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.645700    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.647317 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.647295    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.684784 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.684749    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2tr\" (UniqueName: \"kubernetes.io/projected/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-kube-api-access-8q2tr\") pod \"maas-api-revoke-sub-85237e45-nw5vf\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:16.758184 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:16.758078    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:17.080694 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:03:17.080659    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ab3a05_ef76_4adf_8eec_d92101fda8ee.slice/crio-437ca7d94a8eb55a84683642e5a3e2302337ac966c8f82e07a9316688a1f3154 WatchSource:0}: Error finding container 437ca7d94a8eb55a84683642e5a3e2302337ac966c8f82e07a9316688a1f3154: Status 404 returned error can't find the container with id 437ca7d94a8eb55a84683642e5a3e2302337ac966c8f82e07a9316688a1f3154
Oct 29 23:03:17.081304 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:17.081279    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"]
Oct 29 23:03:17.319520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:17.319483    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf" event={"ID":"e3ab3a05-ef76-4adf-8eec-d92101fda8ee","Type":"ContainerStarted","Data":"437ca7d94a8eb55a84683642e5a3e2302337ac966c8f82e07a9316688a1f3154"}
Oct 29 23:03:18.327371 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:18.327335    4839 generic.go:356] "Generic (PLEG): container finished" podID="e3ab3a05-ef76-4adf-8eec-d92101fda8ee" containerID="e35e065a548b8e86cc785b0119a3a6b937bbf2075c2ad79f7846b87b6f7ece86" exitCode=0
Oct 29 23:03:18.327749 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:18.327379    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf" event={"ID":"e3ab3a05-ef76-4adf-8eec-d92101fda8ee","Type":"ContainerDied","Data":"e35e065a548b8e86cc785b0119a3a6b937bbf2075c2ad79f7846b87b6f7ece86"}
Oct 29 23:03:19.463836 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.463805    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:19.570498 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.570463    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-kube-api-access-8q2tr\" (UniqueName: \"kubernetes.io/projected/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-kube-api-access-8q2tr\") pod \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") "
Oct 29 23:03:19.570676 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.570582    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-xks-ca\") pod \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") "
Oct 29 23:03:19.570676 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.570640    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-openshift-ca\") pod \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\" (UID: \"e3ab3a05-ef76-4adf-8eec-d92101fda8ee\") "
Oct 29 23:03:19.571004 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.570976    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-openshift-ca" pod "e3ab3a05-ef76-4adf-8eec-d92101fda8ee" (UID: "e3ab3a05-ef76-4adf-8eec-d92101fda8ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:03:19.572633 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.572606    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-xks-ca" pod "e3ab3a05-ef76-4adf-8eec-d92101fda8ee" (UID: "e3ab3a05-ef76-4adf-8eec-d92101fda8ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:03:19.572749 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.572635    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-kube-api-access-8q2tr" pod "e3ab3a05-ef76-4adf-8eec-d92101fda8ee" (UID: "e3ab3a05-ef76-4adf-8eec-d92101fda8ee"). InnerVolumeSpecName "kube-api-access-8q2tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:03:19.672137 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.672060    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.672137 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.672088    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:03:19.672137 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:19.672098    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2tr\" (UniqueName: \"kubernetes.io/projected/e3ab3a05-ef76-4adf-8eec-d92101fda8ee-kube-api-access-8q2tr\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:03:20.340061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:20.340020    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf" event={"ID":"e3ab3a05-ef76-4adf-8eec-d92101fda8ee","Type":"ContainerDied","Data":"437ca7d94a8eb55a84683642e5a3e2302337ac966c8f82e07a9316688a1f3154"}
Oct 29 23:03:20.340061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:20.340046    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"
Oct 29 23:03:20.340296 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:20.340099    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="437ca7d94a8eb55a84683642e5a3e2302337ac966c8f82e07a9316688a1f3154"
Oct 29 23:03:21.030608 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:21.030566    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"]
Oct 29 23:03:21.043988 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:21.043956    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b5c2efd-b7plc"]
Oct 29 23:03:23.007636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.007604    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738e6ea5-7781-4fb9-a474-bba91c3906ab" path="/var/lib/kubelet/pods/738e6ea5-7781-4fb9-a474-bba91c3906ab/volumes"
Oct 29 23:03:23.029308 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.029277    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"]
Oct 29 23:03:23.042824 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.042793    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-806a7c55-4mkqz"]
Oct 29 23:03:23.840554 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.840522    4839 scope.go:122] "RemoveContainer" containerID="7915112f2e232ce52b6af4267398159dd1485780598390ec7310bcff84accad0"
Oct 29 23:03:23.849729 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.849704    4839 scope.go:122] "RemoveContainer" containerID="cb9c4c8a61ba170360e0d51e9790fbb145e96edc0509078a197b29f1d3aced8c"
Oct 29 23:03:23.858258 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.858236    4839 scope.go:122] "RemoveContainer" containerID="6633b1d28c3bd8880e1e869b166782f5ded1b4274519be73387583bb988a53d8"
Oct 29 23:03:23.866801 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.866781    4839 scope.go:122] "RemoveContainer" containerID="0935b5814e3e7751756bddbdc8b9c652f629f4db1aeb49637ede51fb456d4061"
Oct 29 23:03:23.875142 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.875120    4839 scope.go:122] "RemoveContainer" containerID="442ea113b62fe009bde3af2f7b7d655a0468257edb7a88d9baee751f368fff4b"
Oct 29 23:03:23.883208 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:23.883184    4839 scope.go:122] "RemoveContainer" containerID="3304443c26b53c5fadd661ee5c285e2e7e311f8aeff5c714f9346bbdf08c3b1a"
Oct 29 23:03:25.007009 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:25.006974    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db9a696-83e2-4db4-9d29-b02eefb561c8" path="/var/lib/kubelet/pods/1db9a696-83e2-4db4-9d29-b02eefb561c8/volumes"
Oct 29 23:03:27.934642 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:27.934603    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:30.415710 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:30.415676    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:03:31.029184 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:31.029130    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"]
Oct 29 23:03:31.042785 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:31.042752    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d0ab918-fxh6s"]
Oct 29 23:03:33.007845 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:33.007811    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41ba44f-c6c6-4e64-95d7-dafec8499831" path="/var/lib/kubelet/pods/c41ba44f-c6c6-4e64-95d7-dafec8499831/volumes"
Oct 29 23:03:33.238667 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:03:33.238624    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:03:36.028563 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:36.028529    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"]
Oct 29 23:03:36.042159 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:36.042124    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5eb311b-brrkc"]
Oct 29 23:03:37.006858 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:03:37.006819    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ce1bbb-d479-4934-b7c9-8d3e128a06cc" path="/var/lib/kubelet/pods/42ce1bbb-d479-4934-b7c9-8d3e128a06cc/volumes"
Oct 29 23:04:03.320852 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:04:03.320810    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:04:07.467284 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467205    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"]
Oct 29 23:04:07.467659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467518    4839 cpu_manager.go:414] "containerMap: removing container" podUID="e3ab3a05-ef76-4adf-8eec-d92101fda8ee" containerName="revoke-keys"
Oct 29 23:04:07.467659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467529    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ab3a05-ef76-4adf-8eec-d92101fda8ee" containerName="revoke-keys"
Oct 29 23:04:07.467659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467553    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy"
Oct 29 23:04:07.467659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467558    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy"
Oct 29 23:04:07.467659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467614    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c17881-8e31-4bc4-a55e-6005c5225912" containerName="istio-proxy"
Oct 29 23:04:07.467659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467621    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ab3a05-ef76-4adf-8eec-d92101fda8ee" containerName="revoke-keys"
Oct 29 23:04:07.467659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.467629    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41b48a5-2064-4e12-87b5-4c8b3fb8694c" containerName="istio-proxy"
Oct 29 23:04:07.470820 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.470804    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.487040 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.487011    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:04:07.487244 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.487190    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:04:07.498829 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.498795    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.498964 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.498909    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nsdg\" (UniqueName: \"kubernetes.io/projected/216f56fa-6413-4a3c-b75a-ff3a201a60ab-kube-api-access-7nsdg\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.498964 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.498948    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.500681 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.500657    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:04:07.500788 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.500743    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:04:07.502640 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.502621    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"]
Oct 29 23:04:07.600277 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.600233    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.600491 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.600312    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nsdg\" (UniqueName: \"kubernetes.io/projected/216f56fa-6413-4a3c-b75a-ff3a201a60ab-kube-api-access-7nsdg\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.600491 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.600333    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.600963 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.600935    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.602911 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.602885    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.621134 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.621110    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nsdg\" (UniqueName: \"kubernetes.io/projected/216f56fa-6413-4a3c-b75a-ff3a201a60ab-kube-api-access-7nsdg\") pod \"maas-api-revoke-sub-c1e271a9-bgn5k\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:07.783575 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:07.783531    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:08.076844 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:04:08.076806    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216f56fa_6413_4a3c_b75a_ff3a201a60ab.slice/crio-c15fefa112287df6613f7bac18383f6853f68ace7ddf1f1b9097824f1620cfcc WatchSource:0}: Error finding container c15fefa112287df6613f7bac18383f6853f68ace7ddf1f1b9097824f1620cfcc: Status 404 returned error can't find the container with id c15fefa112287df6613f7bac18383f6853f68ace7ddf1f1b9097824f1620cfcc
Oct 29 23:04:08.076970 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:08.076907    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"]
Oct 29 23:04:08.612856 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:08.612820    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k" event={"ID":"216f56fa-6413-4a3c-b75a-ff3a201a60ab","Type":"ContainerStarted","Data":"a342393d88c44b3eefc06d9086e1fb91375eae7e313ffcdec4459d77e602dab2"}
Oct 29 23:04:08.613263 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:08.612870    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k" event={"ID":"216f56fa-6413-4a3c-b75a-ff3a201a60ab","Type":"ContainerStarted","Data":"c15fefa112287df6613f7bac18383f6853f68ace7ddf1f1b9097824f1620cfcc"}
Oct 29 23:04:08.654119 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:08.654066    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k" podStartSLOduration=1.654051307 podStartE2EDuration="1.654051307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:08.653132408 +0000 UTC m=+1305.756208679" watchObservedRunningTime="2026-10-29 23:04:08.654051307 +0000 UTC m=+1305.757127577"
Oct 29 23:04:09.625272 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:09.625232    4839 generic.go:356] "Generic (PLEG): container finished" podID="216f56fa-6413-4a3c-b75a-ff3a201a60ab" containerID="a342393d88c44b3eefc06d9086e1fb91375eae7e313ffcdec4459d77e602dab2" exitCode=0
Oct 29 23:04:09.625678 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:09.625312    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k" event={"ID":"216f56fa-6413-4a3c-b75a-ff3a201a60ab","Type":"ContainerDied","Data":"a342393d88c44b3eefc06d9086e1fb91375eae7e313ffcdec4459d77e602dab2"}
Oct 29 23:04:10.763471 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.763447    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:10.828918 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.828889    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-openshift-ca\") pod \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") "
Oct 29 23:04:10.829085 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.828949    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216f56fa-6413-4a3c-b75a-ff3a201a60ab-kube-api-access-7nsdg\" (UniqueName: \"kubernetes.io/projected/216f56fa-6413-4a3c-b75a-ff3a201a60ab-kube-api-access-7nsdg\") pod \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") "
Oct 29 23:04:10.829085 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.829037    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-xks-ca\") pod \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\" (UID: \"216f56fa-6413-4a3c-b75a-ff3a201a60ab\") "
Oct 29 23:04:10.829404 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.829325    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-openshift-ca" pod "216f56fa-6413-4a3c-b75a-ff3a201a60ab" (UID: "216f56fa-6413-4a3c-b75a-ff3a201a60ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:10.831122 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.831091    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-xks-ca" pod "216f56fa-6413-4a3c-b75a-ff3a201a60ab" (UID: "216f56fa-6413-4a3c-b75a-ff3a201a60ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:10.831232 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.831130    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f56fa-6413-4a3c-b75a-ff3a201a60ab-kube-api-access-7nsdg" pod "216f56fa-6413-4a3c-b75a-ff3a201a60ab" (UID: "216f56fa-6413-4a3c-b75a-ff3a201a60ab"). InnerVolumeSpecName "kube-api-access-7nsdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:10.930382 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.930295    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:10.930382 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.930324    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nsdg\" (UniqueName: \"kubernetes.io/projected/216f56fa-6413-4a3c-b75a-ff3a201a60ab-kube-api-access-7nsdg\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:10.930382 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:10.930339    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/216f56fa-6413-4a3c-b75a-ff3a201a60ab-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:11.637998 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:11.637965    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"
Oct 29 23:04:11.638175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:11.638004    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k" event={"ID":"216f56fa-6413-4a3c-b75a-ff3a201a60ab","Type":"ContainerDied","Data":"c15fefa112287df6613f7bac18383f6853f68ace7ddf1f1b9097824f1620cfcc"}
Oct 29 23:04:11.638175 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:11.638050    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c15fefa112287df6613f7bac18383f6853f68ace7ddf1f1b9097824f1620cfcc"
Oct 29 23:04:16.102352 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:16.102311    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:16.203134 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:16.203101    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:04:17.030842 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:17.030804    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"]
Oct 29 23:04:17.045053 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:17.045019    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-761cf924-mmsgw"]
Oct 29 23:04:19.006968 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:19.006938    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a40cb3c-ae14-41de-a4eb-2caa9b391428" path="/var/lib/kubelet/pods/4a40cb3c-ae14-41de-a4eb-2caa9b391428/volumes"
Oct 29 23:04:23.954160 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:23.954092    4839 scope.go:122] "RemoveContainer" containerID="96375467b597b995362f7f0963ccf01d269c71b84ae5708f7cbc98a091efe766"
Oct 29 23:04:23.966887 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:23.966855    4839 scope.go:122] "RemoveContainer" containerID="911a2d0c3197f7563c3814a60323cb3537955235f02a80563eebfe1a06d46e1d"
Oct 29 23:04:23.979275 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:23.979248    4839 scope.go:122] "RemoveContainer" containerID="bf020bb67a26315d7f4879aa965b2d233c7b0b342eb9dcd6574373df7be32a0f"
Oct 29 23:04:23.991615 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:23.991574    4839 scope.go:122] "RemoveContainer" containerID="7cd9e4d8b6a078056672b610d2a2de5c584a7698bc5c530c2174290ef1c43e2f"
Oct 29 23:04:24.003839 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:24.003712    4839 scope.go:122] "RemoveContainer" containerID="087262b2d8e3ddca7eb4525b89c8363d543347de50b922e06e1128a773c58585"
Oct 29 23:04:24.015361 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:24.015249    4839 scope.go:122] "RemoveContainer" containerID="52850aeabeef7586220eeeac6106433cbe8fb5f6daa4be3d4ea18e43fbcb7d7b"
Oct 29 23:04:27.537763 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.537724    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"]
Oct 29 23:04:27.538242 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.538226    4839 cpu_manager.go:414] "containerMap: removing container" podUID="216f56fa-6413-4a3c-b75a-ff3a201a60ab" containerName="revoke-keys"
Oct 29 23:04:27.538297 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.538244    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216f56fa-6413-4a3c-b75a-ff3a201a60ab" containerName="revoke-keys"
Oct 29 23:04:27.538347 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.538336    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="216f56fa-6413-4a3c-b75a-ff3a201a60ab" containerName="revoke-keys"
Oct 29 23:04:27.543320 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.543297    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.560974 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.560944    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:04:27.562859 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.562828    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"]
Oct 29 23:04:27.573516 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.573463    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:04:27.574570 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.574553    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:04:27.574655 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.574569    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:04:27.604380 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.604349    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.604537 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.604386    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5nt\" (UniqueName: \"kubernetes.io/projected/4fc63429-4232-4f69-98d8-e34fb6c2667c-kube-api-access-gw5nt\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.604537 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.604412    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.705224 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.705188    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.705393 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.705296    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.705393 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.705322    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5nt\" (UniqueName: \"kubernetes.io/projected/4fc63429-4232-4f69-98d8-e34fb6c2667c-kube-api-access-gw5nt\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.705887 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.705858    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.707603 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.707581    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.728143 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.728110    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5nt\" (UniqueName: \"kubernetes.io/projected/4fc63429-4232-4f69-98d8-e34fb6c2667c-kube-api-access-gw5nt\") pod \"maas-api-revoke-sub-2e1f5b5f-qdk76\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:27.856779 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:27.856672    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:28.151116 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:04:28.151081    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc63429_4232_4f69_98d8_e34fb6c2667c.slice/crio-78ceaef2525a945395693961e3896fd39700e754827e0fb98f224780de9ff9c7 WatchSource:0}: Error finding container 78ceaef2525a945395693961e3896fd39700e754827e0fb98f224780de9ff9c7: Status 404 returned error can't find the container with id 78ceaef2525a945395693961e3896fd39700e754827e0fb98f224780de9ff9c7
Oct 29 23:04:28.151388 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:28.151364    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"]
Oct 29 23:04:28.751091 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:28.751002    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76" event={"ID":"4fc63429-4232-4f69-98d8-e34fb6c2667c","Type":"ContainerStarted","Data":"45432271ad4edaa6fd2468cf0f4ed9feb01aafc7367ad3fd62c4af7d2d974ab1"}
Oct 29 23:04:28.751091 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:28.751060    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76" event={"ID":"4fc63429-4232-4f69-98d8-e34fb6c2667c","Type":"ContainerStarted","Data":"78ceaef2525a945395693961e3896fd39700e754827e0fb98f224780de9ff9c7"}
Oct 29 23:04:28.804689 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:28.804637    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76" podStartSLOduration=1.804622904 podStartE2EDuration="1.804622904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:28.804231879 +0000 UTC m=+1325.907308162" watchObservedRunningTime="2026-10-29 23:04:28.804622904 +0000 UTC m=+1325.907699174"
Oct 29 23:04:29.759052 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:29.759010    4839 generic.go:356] "Generic (PLEG): container finished" podID="4fc63429-4232-4f69-98d8-e34fb6c2667c" containerID="45432271ad4edaa6fd2468cf0f4ed9feb01aafc7367ad3fd62c4af7d2d974ab1" exitCode=0
Oct 29 23:04:29.759494 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:29.759094    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76" event={"ID":"4fc63429-4232-4f69-98d8-e34fb6c2667c","Type":"ContainerDied","Data":"45432271ad4edaa6fd2468cf0f4ed9feb01aafc7367ad3fd62c4af7d2d974ab1"}
Oct 29 23:04:30.893875 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:30.893849    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:31.035155 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.035127    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-xks-ca\") pod \"4fc63429-4232-4f69-98d8-e34fb6c2667c\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") "
Oct 29 23:04:31.035323 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.035220    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc63429-4232-4f69-98d8-e34fb6c2667c-kube-api-access-gw5nt\" (UniqueName: \"kubernetes.io/projected/4fc63429-4232-4f69-98d8-e34fb6c2667c-kube-api-access-gw5nt\") pod \"4fc63429-4232-4f69-98d8-e34fb6c2667c\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") "
Oct 29 23:04:31.035323 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.035274    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-openshift-ca\") pod \"4fc63429-4232-4f69-98d8-e34fb6c2667c\" (UID: \"4fc63429-4232-4f69-98d8-e34fb6c2667c\") "
Oct 29 23:04:31.035640 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.035617    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-openshift-ca" pod "4fc63429-4232-4f69-98d8-e34fb6c2667c" (UID: "4fc63429-4232-4f69-98d8-e34fb6c2667c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:31.037318 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.037294    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-xks-ca" pod "4fc63429-4232-4f69-98d8-e34fb6c2667c" (UID: "4fc63429-4232-4f69-98d8-e34fb6c2667c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:31.037433 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.037399    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc63429-4232-4f69-98d8-e34fb6c2667c-kube-api-access-gw5nt" pod "4fc63429-4232-4f69-98d8-e34fb6c2667c" (UID: "4fc63429-4232-4f69-98d8-e34fb6c2667c"). InnerVolumeSpecName "kube-api-access-gw5nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:31.136762 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.136733    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:31.136762 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.136758    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fc63429-4232-4f69-98d8-e34fb6c2667c-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:31.136762 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.136770    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5nt\" (UniqueName: \"kubernetes.io/projected/4fc63429-4232-4f69-98d8-e34fb6c2667c-kube-api-access-gw5nt\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:31.775636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.775608    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"
Oct 29 23:04:31.775636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.775623    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76" event={"ID":"4fc63429-4232-4f69-98d8-e34fb6c2667c","Type":"ContainerDied","Data":"78ceaef2525a945395693961e3896fd39700e754827e0fb98f224780de9ff9c7"}
Oct 29 23:04:31.775852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:31.775688    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78ceaef2525a945395693961e3896fd39700e754827e0fb98f224780de9ff9c7"
Oct 29 23:04:33.237804 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:04:33.237767    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:04:38.794733 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.794684    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"]
Oct 29 23:04:38.795464 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.795031    4839 cpu_manager.go:414] "containerMap: removing container" podUID="4fc63429-4232-4f69-98d8-e34fb6c2667c" containerName="revoke-keys"
Oct 29 23:04:38.795464 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.795042    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc63429-4232-4f69-98d8-e34fb6c2667c" containerName="revoke-keys"
Oct 29 23:04:38.795464 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.795103    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc63429-4232-4f69-98d8-e34fb6c2667c" containerName="revoke-keys"
Oct 29 23:04:38.798333 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.798310    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:38.826859 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.826822    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"]
Oct 29 23:04:38.827814 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.827781    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:04:38.827946 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.827782    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:04:38.829074 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.829051    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:04:38.829213 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.829106    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:04:38.904946 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.904912    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:38.905130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.904975    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:38.905130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:38.905094    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z567p\" (UniqueName: \"kubernetes.io/projected/584b3458-109e-4f7a-a384-4aa0fd7799fa-kube-api-access-z567p\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.005663 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.005627    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.005868 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.005687    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.005868 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.005718    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z567p\" (UniqueName: \"kubernetes.io/projected/584b3458-109e-4f7a-a384-4aa0fd7799fa-kube-api-access-z567p\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.006429 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.006403    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.008074 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.008047    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.027103 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.027070    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z567p\" (UniqueName: \"kubernetes.io/projected/584b3458-109e-4f7a-a384-4aa0fd7799fa-kube-api-access-z567p\") pod \"maas-api-revoke-sub-ec843018-74g96\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.114497 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.114405    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:39.449725 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.449693    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"]
Oct 29 23:04:39.449911 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:04:39.449889    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod584b3458_109e_4f7a_a384_4aa0fd7799fa.slice/crio-ed0c60da3b7e57b008e5945f2c9f5a3053a2355efb9702d95c404aa2f396f2d2 WatchSource:0}: Error finding container ed0c60da3b7e57b008e5945f2c9f5a3053a2355efb9702d95c404aa2f396f2d2: Status 404 returned error can't find the container with id ed0c60da3b7e57b008e5945f2c9f5a3053a2355efb9702d95c404aa2f396f2d2
Oct 29 23:04:39.819109 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.819068    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96" event={"ID":"584b3458-109e-4f7a-a384-4aa0fd7799fa","Type":"ContainerStarted","Data":"d68da4befb5514cc1463dd4dce9d883a348fcefb996eefe2bf34b9edea4e0fc8"}
Oct 29 23:04:39.819518 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.819122    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96" event={"ID":"584b3458-109e-4f7a-a384-4aa0fd7799fa","Type":"ContainerStarted","Data":"ed0c60da3b7e57b008e5945f2c9f5a3053a2355efb9702d95c404aa2f396f2d2"}
Oct 29 23:04:39.861458 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:39.861400    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96" podStartSLOduration=1.861383935 podStartE2EDuration="1.861383935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:04:39.85979654 +0000 UTC m=+1336.962872811" watchObservedRunningTime="2026-10-29 23:04:39.861383935 +0000 UTC m=+1336.964460250"
Oct 29 23:04:40.826068 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:40.826031    4839 generic.go:356] "Generic (PLEG): container finished" podID="584b3458-109e-4f7a-a384-4aa0fd7799fa" containerID="d68da4befb5514cc1463dd4dce9d883a348fcefb996eefe2bf34b9edea4e0fc8" exitCode=0
Oct 29 23:04:40.826520 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:40.826116    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96" event={"ID":"584b3458-109e-4f7a-a384-4aa0fd7799fa","Type":"ContainerDied","Data":"d68da4befb5514cc1463dd4dce9d883a348fcefb996eefe2bf34b9edea4e0fc8"}
Oct 29 23:04:41.958761 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:41.958733    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:42.033635 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.033597    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-xks-ca\") pod \"584b3458-109e-4f7a-a384-4aa0fd7799fa\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") "
Oct 29 23:04:42.033799 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.033695    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/584b3458-109e-4f7a-a384-4aa0fd7799fa-kube-api-access-z567p\" (UniqueName: \"kubernetes.io/projected/584b3458-109e-4f7a-a384-4aa0fd7799fa-kube-api-access-z567p\") pod \"584b3458-109e-4f7a-a384-4aa0fd7799fa\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") "
Oct 29 23:04:42.033799 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.033746    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-openshift-ca\") pod \"584b3458-109e-4f7a-a384-4aa0fd7799fa\" (UID: \"584b3458-109e-4f7a-a384-4aa0fd7799fa\") "
Oct 29 23:04:42.034111 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.034086    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-openshift-ca" pod "584b3458-109e-4f7a-a384-4aa0fd7799fa" (UID: "584b3458-109e-4f7a-a384-4aa0fd7799fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:04:42.035750 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.035729    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/584b3458-109e-4f7a-a384-4aa0fd7799fa-kube-api-access-z567p" pod "584b3458-109e-4f7a-a384-4aa0fd7799fa" (UID: "584b3458-109e-4f7a-a384-4aa0fd7799fa"). InnerVolumeSpecName "kube-api-access-z567p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:04:42.035750 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.035739    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-xks-ca" pod "584b3458-109e-4f7a-a384-4aa0fd7799fa" (UID: "584b3458-109e-4f7a-a384-4aa0fd7799fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:04:42.135535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.135455    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.135535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.135482    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z567p\" (UniqueName: \"kubernetes.io/projected/584b3458-109e-4f7a-a384-4aa0fd7799fa-kube-api-access-z567p\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.135535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.135491    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/584b3458-109e-4f7a-a384-4aa0fd7799fa-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:04:42.837692 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.837656    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"
Oct 29 23:04:42.837692 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.837661    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96" event={"ID":"584b3458-109e-4f7a-a384-4aa0fd7799fa","Type":"ContainerDied","Data":"ed0c60da3b7e57b008e5945f2c9f5a3053a2355efb9702d95c404aa2f396f2d2"}
Oct 29 23:04:42.837914 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:04:42.837719    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed0c60da3b7e57b008e5945f2c9f5a3053a2355efb9702d95c404aa2f396f2d2"
Oct 29 23:05:03.237142 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:05:03.237105    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:05:06.554422 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.554373    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"]
Oct 29 23:05:06.554894 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.554874    4839 cpu_manager.go:414] "containerMap: removing container" podUID="584b3458-109e-4f7a-a384-4aa0fd7799fa" containerName="revoke-keys"
Oct 29 23:05:06.554941 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.554898    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="584b3458-109e-4f7a-a384-4aa0fd7799fa" containerName="revoke-keys"
Oct 29 23:05:06.555008 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.554997    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="584b3458-109e-4f7a-a384-4aa0fd7799fa" containerName="revoke-keys"
Oct 29 23:05:06.558618 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.558602    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.574262 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.574227    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"]
Oct 29 23:05:06.585135 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.585108    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:05:06.585329 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.585218    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:05:06.588298 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.588278    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:05:06.588437 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.588366    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:05:06.612865 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.612833    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.612865 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.612869    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gqx\" (UniqueName: \"kubernetes.io/projected/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-kube-api-access-45gqx\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.613061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.612994    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.714295 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.714141    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.714295 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.714212    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gqx\" (UniqueName: \"kubernetes.io/projected/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-kube-api-access-45gqx\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.714295 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.714273    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.719539 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.714923    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.719539 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.716558    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.734851 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.734824    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gqx\" (UniqueName: \"kubernetes.io/projected/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-kube-api-access-45gqx\") pod \"maas-api-revoke-sub-046e1343-hbjtd\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:06.871844 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:06.871755    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:07.163674 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:05:07.163639    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f2e3f5_8cb3_4d17_a99f_9edaf593baa7.slice/crio-d032b8fea4dbab08f61380db48319d9ab721822570dd73fc7dd90bc5c3be9a93 WatchSource:0}: Error finding container d032b8fea4dbab08f61380db48319d9ab721822570dd73fc7dd90bc5c3be9a93: Status 404 returned error can't find the container with id d032b8fea4dbab08f61380db48319d9ab721822570dd73fc7dd90bc5c3be9a93
Oct 29 23:05:07.163869 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:07.163850    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"]
Oct 29 23:05:08.005478 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:08.005444    4839 generic.go:356] "Generic (PLEG): container finished" podID="d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" containerID="500ed6f4eca398047716203fa2f7f48b01c79a9682ea95c876ed562d6370d77a" exitCode=0
Oct 29 23:05:08.005874 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:08.005485    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd" event={"ID":"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7","Type":"ContainerDied","Data":"500ed6f4eca398047716203fa2f7f48b01c79a9682ea95c876ed562d6370d77a"}
Oct 29 23:05:08.005874 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:08.005536    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd" event={"ID":"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7","Type":"ContainerStarted","Data":"d032b8fea4dbab08f61380db48319d9ab721822570dd73fc7dd90bc5c3be9a93"}
Oct 29 23:05:09.147758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.147734    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:09.236017 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.235973    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-openshift-ca\") pod \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") "
Oct 29 23:05:09.236215 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.236077    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-kube-api-access-45gqx\" (UniqueName: \"kubernetes.io/projected/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-kube-api-access-45gqx\") pod \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") "
Oct 29 23:05:09.236215 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.236105    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-xks-ca\") pod \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\" (UID: \"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7\") "
Oct 29 23:05:09.236370 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.236346    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-openshift-ca" pod "d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" (UID: "d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:09.238138 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.238114    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-xks-ca" pod "d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" (UID: "d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:09.238252 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.238176    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-kube-api-access-45gqx" pod "d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" (UID: "d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7"). InnerVolumeSpecName "kube-api-access-45gqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:09.336882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.336845    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.336882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.336877    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45gqx\" (UniqueName: \"kubernetes.io/projected/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-kube-api-access-45gqx\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:09.337080 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:09.336893    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:10.017475 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:10.017434    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd" event={"ID":"d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7","Type":"ContainerDied","Data":"d032b8fea4dbab08f61380db48319d9ab721822570dd73fc7dd90bc5c3be9a93"}
Oct 29 23:05:10.017475 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:10.017464    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"
Oct 29 23:05:10.017691 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:10.017486    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d032b8fea4dbab08f61380db48319d9ab721822570dd73fc7dd90bc5c3be9a93"
Oct 29 23:05:11.027244 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:11.027206    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"]
Oct 29 23:05:11.040930 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:11.040901    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3884fae-xbqm2"]
Oct 29 23:05:13.007163 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:13.007120    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1149292-e0a6-4f28-90c3-5147775bf9ba" path="/var/lib/kubelet/pods/b1149292-e0a6-4f28-90c3-5147775bf9ba/volumes"
Oct 29 23:05:16.235097 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.235057    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:05:16.880516 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.880479    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"]
Oct 29 23:05:16.880840 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.880822    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" containerName="revoke-keys"
Oct 29 23:05:16.880840 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.880836    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" containerName="revoke-keys"
Oct 29 23:05:16.880934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.880886    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" containerName="revoke-keys"
Oct 29 23:05:16.883983 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.883966    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:16.902934 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.902908    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"]
Oct 29 23:05:16.914012 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.913977    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:05:16.914012 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.913994    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:05:16.914267 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.914016    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:05:16.914466 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:16.914451    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:05:17.005884 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.005850    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.006061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.005923    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7fq2\" (UniqueName: \"kubernetes.io/projected/07f2d491-99f5-4220-90a3-7ba792808575-kube-api-access-z7fq2\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.006061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.005988    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.106551 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.106515    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.106551 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.106559    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7fq2\" (UniqueName: \"kubernetes.io/projected/07f2d491-99f5-4220-90a3-7ba792808575-kube-api-access-z7fq2\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.106782 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.106594    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.107332 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.107306    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.109026 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.109005    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.129058 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.129029    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7fq2\" (UniqueName: \"kubernetes.io/projected/07f2d491-99f5-4220-90a3-7ba792808575-kube-api-access-z7fq2\") pod \"maas-api-revoke-sub-d3fba056-q6wwk\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.196658 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.196574    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:17.491221 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:05:17.491184    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f2d491_99f5_4220_90a3_7ba792808575.slice/crio-d0f932b0882c11e0b37763616f09d540de17bfc1e9bdb1b04e6affd03dc9b181 WatchSource:0}: Error finding container d0f932b0882c11e0b37763616f09d540de17bfc1e9bdb1b04e6affd03dc9b181: Status 404 returned error can't find the container with id d0f932b0882c11e0b37763616f09d540de17bfc1e9bdb1b04e6affd03dc9b181
Oct 29 23:05:17.491568 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:17.491248    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"]
Oct 29 23:05:18.061496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:18.061463    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk" event={"ID":"07f2d491-99f5-4220-90a3-7ba792808575","Type":"ContainerStarted","Data":"b73a6967b31df59c458c4a2d9ab61c7c9ce48901ef6c903d2543892559c3a3c2"}
Oct 29 23:05:18.061669 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:18.061514    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk" event={"ID":"07f2d491-99f5-4220-90a3-7ba792808575","Type":"ContainerStarted","Data":"d0f932b0882c11e0b37763616f09d540de17bfc1e9bdb1b04e6affd03dc9b181"}
Oct 29 23:05:18.113275 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:18.113203    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk" podStartSLOduration=2.113183882 podStartE2EDuration="2.113183882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:05:18.113009307 +0000 UTC m=+1375.216085579" watchObservedRunningTime="2026-10-29 23:05:18.113183882 +0000 UTC m=+1375.216260151"
Oct 29 23:05:19.093171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:19.089447    4839 generic.go:356] "Generic (PLEG): container finished" podID="07f2d491-99f5-4220-90a3-7ba792808575" containerID="b73a6967b31df59c458c4a2d9ab61c7c9ce48901ef6c903d2543892559c3a3c2" exitCode=0
Oct 29 23:05:19.093171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:19.089551    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk" event={"ID":"07f2d491-99f5-4220-90a3-7ba792808575","Type":"ContainerDied","Data":"b73a6967b31df59c458c4a2d9ab61c7c9ce48901ef6c903d2543892559c3a3c2"}
Oct 29 23:05:20.260182 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:20.259618    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:05:20.260182 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:20.259730    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:05:20.858254 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:20.858182    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:21.024681 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.024653    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f2d491-99f5-4220-90a3-7ba792808575-kube-api-access-z7fq2\" (UniqueName: \"kubernetes.io/projected/07f2d491-99f5-4220-90a3-7ba792808575-kube-api-access-z7fq2\") pod \"07f2d491-99f5-4220-90a3-7ba792808575\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") "
Oct 29 23:05:21.024850 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.024781    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-xks-ca\") pod \"07f2d491-99f5-4220-90a3-7ba792808575\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") "
Oct 29 23:05:21.024850 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.024811    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-openshift-ca\") pod \"07f2d491-99f5-4220-90a3-7ba792808575\" (UID: \"07f2d491-99f5-4220-90a3-7ba792808575\") "
Oct 29 23:05:21.025231 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.025199    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-openshift-ca" pod "07f2d491-99f5-4220-90a3-7ba792808575" (UID: "07f2d491-99f5-4220-90a3-7ba792808575"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:21.026815 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.026792    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f2d491-99f5-4220-90a3-7ba792808575-kube-api-access-z7fq2" pod "07f2d491-99f5-4220-90a3-7ba792808575" (UID: "07f2d491-99f5-4220-90a3-7ba792808575"). InnerVolumeSpecName "kube-api-access-z7fq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:21.026885 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.026820    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-xks-ca" pod "07f2d491-99f5-4220-90a3-7ba792808575" (UID: "07f2d491-99f5-4220-90a3-7ba792808575"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:21.111601 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.111520    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"
Oct 29 23:05:21.111601 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.111560    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk" event={"ID":"07f2d491-99f5-4220-90a3-7ba792808575","Type":"ContainerDied","Data":"d0f932b0882c11e0b37763616f09d540de17bfc1e9bdb1b04e6affd03dc9b181"}
Oct 29 23:05:21.111803 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.111622    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0f932b0882c11e0b37763616f09d540de17bfc1e9bdb1b04e6affd03dc9b181"
Oct 29 23:05:21.126392 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.126363    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:21.126392 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.126394    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f2d491-99f5-4220-90a3-7ba792808575-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:21.126591 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:21.126409    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7fq2\" (UniqueName: \"kubernetes.io/projected/07f2d491-99f5-4220-90a3-7ba792808575-kube-api-access-z7fq2\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:24.077699 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:24.077661    4839 scope.go:122] "RemoveContainer" containerID="6f82df9ac2af3f2a7217a6d5fe6d5b242f55cfcc3e25738d36676af00af69f98"
Oct 29 23:05:24.086889 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:24.086868    4839 scope.go:122] "RemoveContainer" containerID="50c05fa09a583a3fb9b4b3e1788eb5aa7a66f8c5821c46500c2443fdd1ddcc82"
Oct 29 23:05:24.094965 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:24.094946    4839 scope.go:122] "RemoveContainer" containerID="4ae1e6a27d00a63e3eb83adf0251e32553dd4c5e2a8826ceea0254d00a30473f"
Oct 29 23:05:24.102798 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:24.102779    4839 scope.go:122] "RemoveContainer" containerID="198759c8d6ec73d9b5531869b097dcbd2d1235d6ca69179e68f55bcfa845db69"
Oct 29 23:05:24.110517 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:24.110495    4839 scope.go:122] "RemoveContainer" containerID="d0fb0b52883cec945ba4c2650e1cbff591635c6796adc763554bed51fae2b08d"
Oct 29 23:05:33.250692 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:05:33.250614    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:05:34.911173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:34.909253    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:05:34.911173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:34.909318    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:05:35.220230 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:35.219277    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:05:35.220230 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:35.219366    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:05:38.552421 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.552383    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"]
Oct 29 23:05:38.553083 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.552733    4839 cpu_manager.go:414] "containerMap: removing container" podUID="07f2d491-99f5-4220-90a3-7ba792808575" containerName="revoke-keys"
Oct 29 23:05:38.553083 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.552744    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f2d491-99f5-4220-90a3-7ba792808575" containerName="revoke-keys"
Oct 29 23:05:38.553083 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.552811    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f2d491-99f5-4220-90a3-7ba792808575" containerName="revoke-keys"
Oct 29 23:05:38.557832 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.557803    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.575713 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.575682    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"]
Oct 29 23:05:38.585911 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.585879    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:05:38.585911 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.585879    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:05:38.586095 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.585921    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:05:38.588094 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.588072    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:05:38.624300 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.624261    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.624468 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.624329    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.624468 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.624424    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqkw\" (UniqueName: \"kubernetes.io/projected/b942b226-bc27-446f-bffc-8fc8aed9acff-kube-api-access-prqkw\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.725203 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.725136    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.725386 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.725238    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.725386 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.725278    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqkw\" (UniqueName: \"kubernetes.io/projected/b942b226-bc27-446f-bffc-8fc8aed9acff-kube-api-access-prqkw\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.725914 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.725887    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.727476 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.727453    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.745708 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.745681    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqkw\" (UniqueName: \"kubernetes.io/projected/b942b226-bc27-446f-bffc-8fc8aed9acff-kube-api-access-prqkw\") pod \"maas-api-revoke-sub-ae594979-7l9dj\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:38.874307 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:38.874213    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:39.168235 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:05:39.168204    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb942b226_bc27_446f_bffc_8fc8aed9acff.slice/crio-e8b6d2d81b06037f05f4f5e2c8b603a8a6e854df19ff943c8911e10d818750f2 WatchSource:0}: Error finding container e8b6d2d81b06037f05f4f5e2c8b603a8a6e854df19ff943c8911e10d818750f2: Status 404 returned error can't find the container with id e8b6d2d81b06037f05f4f5e2c8b603a8a6e854df19ff943c8911e10d818750f2
Oct 29 23:05:39.169015 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:39.168991    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"]
Oct 29 23:05:39.275324 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:39.275289    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj" event={"ID":"b942b226-bc27-446f-bffc-8fc8aed9acff","Type":"ContainerStarted","Data":"e8b6d2d81b06037f05f4f5e2c8b603a8a6e854df19ff943c8911e10d818750f2"}
Oct 29 23:05:40.281861 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:40.281827    4839 generic.go:356] "Generic (PLEG): container finished" podID="b942b226-bc27-446f-bffc-8fc8aed9acff" containerID="0ca764c02430ac96ddc089724b417e76ae151efb395db10e7525c4aed5d43647" exitCode=0
Oct 29 23:05:40.282341 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:40.281922    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj" event={"ID":"b942b226-bc27-446f-bffc-8fc8aed9acff","Type":"ContainerDied","Data":"0ca764c02430ac96ddc089724b417e76ae151efb395db10e7525c4aed5d43647"}
Oct 29 23:05:41.415472 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.415440    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:41.449877 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.449848    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-openshift-ca\") pod \"b942b226-bc27-446f-bffc-8fc8aed9acff\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") "
Oct 29 23:05:41.450019 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.449886    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-xks-ca\") pod \"b942b226-bc27-446f-bffc-8fc8aed9acff\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") "
Oct 29 23:05:41.450019 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.449940    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b942b226-bc27-446f-bffc-8fc8aed9acff-kube-api-access-prqkw\" (UniqueName: \"kubernetes.io/projected/b942b226-bc27-446f-bffc-8fc8aed9acff-kube-api-access-prqkw\") pod \"b942b226-bc27-446f-bffc-8fc8aed9acff\" (UID: \"b942b226-bc27-446f-bffc-8fc8aed9acff\") "
Oct 29 23:05:41.450238 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.450205    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-openshift-ca" pod "b942b226-bc27-446f-bffc-8fc8aed9acff" (UID: "b942b226-bc27-446f-bffc-8fc8aed9acff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:05:41.451961 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.451934    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b942b226-bc27-446f-bffc-8fc8aed9acff-kube-api-access-prqkw" pod "b942b226-bc27-446f-bffc-8fc8aed9acff" (UID: "b942b226-bc27-446f-bffc-8fc8aed9acff"). InnerVolumeSpecName "kube-api-access-prqkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:05:41.452058 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.452022    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-xks-ca" pod "b942b226-bc27-446f-bffc-8fc8aed9acff" (UID: "b942b226-bc27-446f-bffc-8fc8aed9acff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:05:41.550847 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.550762    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:41.550847 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.550792    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b942b226-bc27-446f-bffc-8fc8aed9acff-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:41.550847 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:41.550803    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prqkw\" (UniqueName: \"kubernetes.io/projected/b942b226-bc27-446f-bffc-8fc8aed9acff-kube-api-access-prqkw\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:05:42.295004 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:42.294970    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"
Oct 29 23:05:42.295214 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:42.295003    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj" event={"ID":"b942b226-bc27-446f-bffc-8fc8aed9acff","Type":"ContainerDied","Data":"e8b6d2d81b06037f05f4f5e2c8b603a8a6e854df19ff943c8911e10d818750f2"}
Oct 29 23:05:42.295214 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:42.295077    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8b6d2d81b06037f05f4f5e2c8b603a8a6e854df19ff943c8911e10d818750f2"
Oct 29 23:05:45.030085 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:45.030053    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"]
Oct 29 23:05:45.043970 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:45.043939    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"]
Oct 29 23:05:45.045703 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:45.045685    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f371fca0-r6rn8"]
Oct 29 23:05:45.047396 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:45.047376    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-596865a7-8fldr"]
Oct 29 23:05:47.007375 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:47.007343    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b67e9a89-ed82-4e56-93a7-73e5518c4d96" path="/var/lib/kubelet/pods/b67e9a89-ed82-4e56-93a7-73e5518c4d96/volumes"
Oct 29 23:05:47.007762 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:47.007735    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fd88ed-b825-4be1-8489-a0a383eba610" path="/var/lib/kubelet/pods/c2fd88ed-b825-4be1-8489-a0a383eba610/volumes"
Oct 29 23:05:49.030970 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:49.030932    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"]
Oct 29 23:05:49.044444 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:49.044414    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98d8025f-s8vq6"]
Oct 29 23:05:51.006903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:05:51.006870    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8af9217-f7d5-4aa6-acc2-58f9aef83b6e" path="/var/lib/kubelet/pods/a8af9217-f7d5-4aa6-acc2-58f9aef83b6e/volumes"
Oct 29 23:06:00.035361 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.035324    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"]
Oct 29 23:06:00.035751 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.035664    4839 cpu_manager.go:414] "containerMap: removing container" podUID="b942b226-bc27-446f-bffc-8fc8aed9acff" containerName="revoke-keys"
Oct 29 23:06:00.035751 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.035674    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b942b226-bc27-446f-bffc-8fc8aed9acff" containerName="revoke-keys"
Oct 29 23:06:00.035751 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.035731    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="b942b226-bc27-446f-bffc-8fc8aed9acff" containerName="revoke-keys"
Oct 29 23:06:00.039873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.039855    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.061857 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.061822    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"]
Oct 29 23:06:00.072487 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.072456    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:06:00.072487 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.072481    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:06:00.072850 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.072529    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:06:00.074315 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.074295    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:06:00.149650 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.149607    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8gf\" (UniqueName: \"kubernetes.io/projected/ac92e5cd-f044-4c43-b04e-243220fc1474-kube-api-access-2t8gf\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.149829 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.149671    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.149829 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.149763    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.250793 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.250758    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8gf\" (UniqueName: \"kubernetes.io/projected/ac92e5cd-f044-4c43-b04e-243220fc1474-kube-api-access-2t8gf\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.250964 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.250805    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.250964 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.250839    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.251513 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.251491    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.253108 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.253091    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.272363 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.272332    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8gf\" (UniqueName: \"kubernetes.io/projected/ac92e5cd-f044-4c43-b04e-243220fc1474-kube-api-access-2t8gf\") pod \"maas-api-revoke-sub-a4aef7d7-t5l2x\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.353532 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.353453    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:00.648081 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:00.648052    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"]
Oct 29 23:06:00.648223 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:06:00.648081    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac92e5cd_f044_4c43_b04e_243220fc1474.slice/crio-387afab7a76514591ef84d1fd3b4c2617bcbb323ab8a87691e31325d2f6a2c6e WatchSource:0}: Error finding container 387afab7a76514591ef84d1fd3b4c2617bcbb323ab8a87691e31325d2f6a2c6e: Status 404 returned error can't find the container with id 387afab7a76514591ef84d1fd3b4c2617bcbb323ab8a87691e31325d2f6a2c6e
Oct 29 23:06:01.442442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:01.442407    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x" event={"ID":"ac92e5cd-f044-4c43-b04e-243220fc1474","Type":"ContainerStarted","Data":"d49d4069c886765a20bdec4e9a3e8cf7ce44b54471f5fd1d4e428b7de0d9236e"}
Oct 29 23:06:01.442837 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:01.442466    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x" event={"ID":"ac92e5cd-f044-4c43-b04e-243220fc1474","Type":"ContainerStarted","Data":"387afab7a76514591ef84d1fd3b4c2617bcbb323ab8a87691e31325d2f6a2c6e"}
Oct 29 23:06:01.486030 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:01.485976    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x" podStartSLOduration=1.48596075 podStartE2EDuration="1.48596075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:06:01.484081781 +0000 UTC m=+1418.587158052" watchObservedRunningTime="2026-10-29 23:06:01.48596075 +0000 UTC m=+1418.589037022"
Oct 29 23:06:02.449494 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:02.449459    4839 generic.go:356] "Generic (PLEG): container finished" podID="ac92e5cd-f044-4c43-b04e-243220fc1474" containerID="d49d4069c886765a20bdec4e9a3e8cf7ce44b54471f5fd1d4e428b7de0d9236e" exitCode=0
Oct 29 23:06:02.449893 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:02.449549    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x" event={"ID":"ac92e5cd-f044-4c43-b04e-243220fc1474","Type":"ContainerDied","Data":"d49d4069c886765a20bdec4e9a3e8cf7ce44b54471f5fd1d4e428b7de0d9236e"}
Oct 29 23:06:03.241138 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:06:03.241100    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:06:03.586063 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.586034    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:03.682912 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.682874    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac92e5cd-f044-4c43-b04e-243220fc1474-kube-api-access-2t8gf\" (UniqueName: \"kubernetes.io/projected/ac92e5cd-f044-4c43-b04e-243220fc1474-kube-api-access-2t8gf\") pod \"ac92e5cd-f044-4c43-b04e-243220fc1474\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") "
Oct 29 23:06:03.683087 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.682958    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-xks-ca\") pod \"ac92e5cd-f044-4c43-b04e-243220fc1474\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") "
Oct 29 23:06:03.683087 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.682995    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-openshift-ca\") pod \"ac92e5cd-f044-4c43-b04e-243220fc1474\" (UID: \"ac92e5cd-f044-4c43-b04e-243220fc1474\") "
Oct 29 23:06:03.683445 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.683417    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-openshift-ca" pod "ac92e5cd-f044-4c43-b04e-243220fc1474" (UID: "ac92e5cd-f044-4c43-b04e-243220fc1474"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:03.685006 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.684970    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-xks-ca" pod "ac92e5cd-f044-4c43-b04e-243220fc1474" (UID: "ac92e5cd-f044-4c43-b04e-243220fc1474"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:03.685100 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.685031    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac92e5cd-f044-4c43-b04e-243220fc1474-kube-api-access-2t8gf" pod "ac92e5cd-f044-4c43-b04e-243220fc1474" (UID: "ac92e5cd-f044-4c43-b04e-243220fc1474"). InnerVolumeSpecName "kube-api-access-2t8gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:03.784395 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.784364    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:03.784395 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.784391    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac92e5cd-f044-4c43-b04e-243220fc1474-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:03.784395 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:03.784401    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t8gf\" (UniqueName: \"kubernetes.io/projected/ac92e5cd-f044-4c43-b04e-243220fc1474-kube-api-access-2t8gf\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:04.460975 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:04.460941    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x" event={"ID":"ac92e5cd-f044-4c43-b04e-243220fc1474","Type":"ContainerDied","Data":"387afab7a76514591ef84d1fd3b4c2617bcbb323ab8a87691e31325d2f6a2c6e"}
Oct 29 23:06:04.461206 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:04.460997    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="387afab7a76514591ef84d1fd3b4c2617bcbb323ab8a87691e31325d2f6a2c6e"
Oct 29 23:06:04.461206 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:04.460957    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"
Oct 29 23:06:16.340324 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:16.340287    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:20.183056 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.183009    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"]
Oct 29 23:06:20.183581 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.183504    4839 cpu_manager.go:414] "containerMap: removing container" podUID="ac92e5cd-f044-4c43-b04e-243220fc1474" containerName="revoke-keys"
Oct 29 23:06:20.183581 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.183524    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac92e5cd-f044-4c43-b04e-243220fc1474" containerName="revoke-keys"
Oct 29 23:06:20.183702 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.183639    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac92e5cd-f044-4c43-b04e-243220fc1474" containerName="revoke-keys"
Oct 29 23:06:20.186828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.186807    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.207574 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.207540    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"]
Oct 29 23:06:20.218502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.218463    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:06:20.218655 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.218539    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:06:20.218829 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.218794    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:06:20.218966 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.218883    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:06:20.265343 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.265310    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfsb\" (UniqueName: \"kubernetes.io/projected/563a1648-8fd8-45b8-a852-a07a4101dc17-kube-api-access-fvfsb\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.265542 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.265386    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.265542 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.265501    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.366739 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.366702    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.366943 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.366860    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.366943 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.366925    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfsb\" (UniqueName: \"kubernetes.io/projected/563a1648-8fd8-45b8-a852-a07a4101dc17-kube-api-access-fvfsb\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.367464 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.367438    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.369217 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.369197    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.388347 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.388312    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfsb\" (UniqueName: \"kubernetes.io/projected/563a1648-8fd8-45b8-a852-a07a4101dc17-kube-api-access-fvfsb\") pod \"maas-api-revoke-sub-79805ca9-smwsk\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.499622 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.499535    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:20.791016 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:20.790987    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"]
Oct 29 23:06:20.791344 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:06:20.791316    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563a1648_8fd8_45b8_a852_a07a4101dc17.slice/crio-19eeb58b06cb72307b0998ed0d0123cc89451c7e18cfdfd0df49ac7c0435e7c2 WatchSource:0}: Error finding container 19eeb58b06cb72307b0998ed0d0123cc89451c7e18cfdfd0df49ac7c0435e7c2: Status 404 returned error can't find the container with id 19eeb58b06cb72307b0998ed0d0123cc89451c7e18cfdfd0df49ac7c0435e7c2
Oct 29 23:06:21.587345 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:21.587310    4839 generic.go:356] "Generic (PLEG): container finished" podID="563a1648-8fd8-45b8-a852-a07a4101dc17" containerID="c42d44835c750c93feb26972824e08063995dbea74ba8b9ca90a440b55676cdb" exitCode=0
Oct 29 23:06:21.587711 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:21.587376    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk" event={"ID":"563a1648-8fd8-45b8-a852-a07a4101dc17","Type":"ContainerDied","Data":"c42d44835c750c93feb26972824e08063995dbea74ba8b9ca90a440b55676cdb"}
Oct 29 23:06:21.587711 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:21.587415    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk" event={"ID":"563a1648-8fd8-45b8-a852-a07a4101dc17","Type":"ContainerStarted","Data":"19eeb58b06cb72307b0998ed0d0123cc89451c7e18cfdfd0df49ac7c0435e7c2"}
Oct 29 23:06:22.718504 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.718480    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:22.790196 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.790167    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/563a1648-8fd8-45b8-a852-a07a4101dc17-kube-api-access-fvfsb\" (UniqueName: \"kubernetes.io/projected/563a1648-8fd8-45b8-a852-a07a4101dc17-kube-api-access-fvfsb\") pod \"563a1648-8fd8-45b8-a852-a07a4101dc17\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") "
Oct 29 23:06:22.790370 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.790300    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-openshift-ca\") pod \"563a1648-8fd8-45b8-a852-a07a4101dc17\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") "
Oct 29 23:06:22.790370 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.790331    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-xks-ca\") pod \"563a1648-8fd8-45b8-a852-a07a4101dc17\" (UID: \"563a1648-8fd8-45b8-a852-a07a4101dc17\") "
Oct 29 23:06:22.790729 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.790699    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-openshift-ca" pod "563a1648-8fd8-45b8-a852-a07a4101dc17" (UID: "563a1648-8fd8-45b8-a852-a07a4101dc17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:22.792465 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.792439    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/563a1648-8fd8-45b8-a852-a07a4101dc17-kube-api-access-fvfsb" pod "563a1648-8fd8-45b8-a852-a07a4101dc17" (UID: "563a1648-8fd8-45b8-a852-a07a4101dc17"). InnerVolumeSpecName "kube-api-access-fvfsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:22.792573 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.792518    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-xks-ca" pod "563a1648-8fd8-45b8-a852-a07a4101dc17" (UID: "563a1648-8fd8-45b8-a852-a07a4101dc17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:22.891595 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.891520    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfsb\" (UniqueName: \"kubernetes.io/projected/563a1648-8fd8-45b8-a852-a07a4101dc17-kube-api-access-fvfsb\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:22.891595 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.891547    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:22.891595 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:22.891560    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/563a1648-8fd8-45b8-a852-a07a4101dc17-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:23.602401 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:23.602363    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"
Oct 29 23:06:23.602401 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:23.602384    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk" event={"ID":"563a1648-8fd8-45b8-a852-a07a4101dc17","Type":"ContainerDied","Data":"19eeb58b06cb72307b0998ed0d0123cc89451c7e18cfdfd0df49ac7c0435e7c2"}
Oct 29 23:06:23.602649 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:23.602433    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19eeb58b06cb72307b0998ed0d0123cc89451c7e18cfdfd0df49ac7c0435e7c2"
Oct 29 23:06:24.156288 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:24.156253    4839 scope.go:122] "RemoveContainer" containerID="0d334f0dc86f461aa2ec786b6e2e5e37d584b184128c28fbaee535f90dd3aef4"
Oct 29 23:06:24.165936 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:24.165917    4839 scope.go:122] "RemoveContainer" containerID="84c258bfcb5ec2673963061a96e711e9144ef6c597f29c2d3e8b0d315e7ed130"
Oct 29 23:06:24.175301 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:24.175278    4839 scope.go:122] "RemoveContainer" containerID="4b232778c3cccdb752d5f435869d1ac081f062ebbe424d6db1d3e4deb965e4bd"
Oct 29 23:06:29.488325 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:29.488285    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:29.582171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:29.582117    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:06:33.237097 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:06:33.237060    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:06:40.241888 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.241849    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"]
Oct 29 23:06:40.242295 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.242206    4839 cpu_manager.go:414] "containerMap: removing container" podUID="563a1648-8fd8-45b8-a852-a07a4101dc17" containerName="revoke-keys"
Oct 29 23:06:40.242295 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.242218    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="563a1648-8fd8-45b8-a852-a07a4101dc17" containerName="revoke-keys"
Oct 29 23:06:40.242295 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.242289    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="563a1648-8fd8-45b8-a852-a07a4101dc17" containerName="revoke-keys"
Oct 29 23:06:40.247082 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.247062    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.265653 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.265621    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"]
Oct 29 23:06:40.276890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.276846    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:06:40.276890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.276853    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:06:40.276890 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.276875    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:06:40.279566 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.279534    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:06:40.375198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.375131    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.375387 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.375255    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttqw\" (UniqueName: \"kubernetes.io/projected/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-kube-api-access-6ttqw\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.375387 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.375300    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.476161 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.476123    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttqw\" (UniqueName: \"kubernetes.io/projected/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-kube-api-access-6ttqw\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.476358 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.476184    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.476358 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.476276    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.476827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.476805    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.478543 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.478523    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.497353 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.497290    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttqw\" (UniqueName: \"kubernetes.io/projected/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-kube-api-access-6ttqw\") pod \"maas-api-revoke-sub-0b7e8edb-ptmkw\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.563199 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.563163    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:40.855484 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:40.855453    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"]
Oct 29 23:06:40.855605 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:06:40.855482    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2917e9a9_0e6d_4b37_b8c2_7f6ac98e6351.slice/crio-68d46cb98afaf5ec8c9c7d0bdeb64c0f1a18fe8a47d39f76e6aada451e47bcff WatchSource:0}: Error finding container 68d46cb98afaf5ec8c9c7d0bdeb64c0f1a18fe8a47d39f76e6aada451e47bcff: Status 404 returned error can't find the container with id 68d46cb98afaf5ec8c9c7d0bdeb64c0f1a18fe8a47d39f76e6aada451e47bcff
Oct 29 23:06:41.764368 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:41.764335    4839 generic.go:356] "Generic (PLEG): container finished" podID="2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" containerID="62c540a16368adef924ac9ea70f42dbc5bbefa159be2a4f5eb2ff2b7461f5270" exitCode=0
Oct 29 23:06:41.764726 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:41.764377    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw" event={"ID":"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351","Type":"ContainerDied","Data":"62c540a16368adef924ac9ea70f42dbc5bbefa159be2a4f5eb2ff2b7461f5270"}
Oct 29 23:06:41.764726 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:41.764421    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw" event={"ID":"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351","Type":"ContainerStarted","Data":"68d46cb98afaf5ec8c9c7d0bdeb64c0f1a18fe8a47d39f76e6aada451e47bcff"}
Oct 29 23:06:42.899653 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:42.899628    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:42.998893 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:42.998861    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-xks-ca\") pod \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") "
Oct 29 23:06:42.999086 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:42.998964    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-kube-api-access-6ttqw\" (UniqueName: \"kubernetes.io/projected/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-kube-api-access-6ttqw\") pod \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") "
Oct 29 23:06:42.999086 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:42.999017    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-openshift-ca\") pod \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\" (UID: \"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351\") "
Oct 29 23:06:42.999407 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:42.999380    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-openshift-ca" pod "2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" (UID: "2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:43.001076 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.001046    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-xks-ca" pod "2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" (UID: "2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:43.001202 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.001176    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-kube-api-access-6ttqw" pod "2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" (UID: "2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351"). InnerVolumeSpecName "kube-api-access-6ttqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:43.100672 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.100592    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:43.100672 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.100621    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:43.100672 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.100632    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ttqw\" (UniqueName: \"kubernetes.io/projected/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351-kube-api-access-6ttqw\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:43.775546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.775514    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"
Oct 29 23:06:43.775723 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.775544    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw" event={"ID":"2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351","Type":"ContainerDied","Data":"68d46cb98afaf5ec8c9c7d0bdeb64c0f1a18fe8a47d39f76e6aada451e47bcff"}
Oct 29 23:06:43.775723 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:43.775600    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d46cb98afaf5ec8c9c7d0bdeb64c0f1a18fe8a47d39f76e6aada451e47bcff"
Oct 29 23:06:51.093182 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.093136    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"]
Oct 29 23:06:51.093684 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.093488    4839 cpu_manager.go:414] "containerMap: removing container" podUID="2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" containerName="revoke-keys"
Oct 29 23:06:51.093684 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.093500    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" containerName="revoke-keys"
Oct 29 23:06:51.093684 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.093566    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" containerName="revoke-keys"
Oct 29 23:06:51.096623 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.096606    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.125194 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.125135    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"]
Oct 29 23:06:51.135542 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.135510    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:06:51.135765 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.135742    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:06:51.135854 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.135792    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:06:51.135919 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.135866    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:06:51.162920 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.162885    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.163102 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.163015    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.163102 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.163062    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfs74\" (UniqueName: \"kubernetes.io/projected/146a6029-9c1d-4eee-8071-4482355322aa-kube-api-access-gfs74\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.264301 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.264255    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.264301 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.264307    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfs74\" (UniqueName: \"kubernetes.io/projected/146a6029-9c1d-4eee-8071-4482355322aa-kube-api-access-gfs74\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.264568 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.264368    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.265049 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.264996    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.266563 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.266542    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.286156 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.286109    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfs74\" (UniqueName: \"kubernetes.io/projected/146a6029-9c1d-4eee-8071-4482355322aa-kube-api-access-gfs74\") pod \"maas-api-revoke-sub-3d4156f1-lsxt8\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.409349 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.409272    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:51.701248 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:06:51.701217    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146a6029_9c1d_4eee_8071_4482355322aa.slice/crio-793e61b9e2f67a4071e66c72e8d602f59ab70fded81840a464b3a81f074e9d97 WatchSource:0}: Error finding container 793e61b9e2f67a4071e66c72e8d602f59ab70fded81840a464b3a81f074e9d97: Status 404 returned error can't find the container with id 793e61b9e2f67a4071e66c72e8d602f59ab70fded81840a464b3a81f074e9d97
Oct 29 23:06:51.702294 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.702267    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"]
Oct 29 23:06:51.818645 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:51.818600    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8" event={"ID":"146a6029-9c1d-4eee-8071-4482355322aa","Type":"ContainerStarted","Data":"793e61b9e2f67a4071e66c72e8d602f59ab70fded81840a464b3a81f074e9d97"}
Oct 29 23:06:52.825584 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:52.825494    4839 generic.go:356] "Generic (PLEG): container finished" podID="146a6029-9c1d-4eee-8071-4482355322aa" containerID="79c94cb22761da13b6913a44dfab20da8009b822e9b9ecf8447f8bbfdd177525" exitCode=0
Oct 29 23:06:52.825584 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:52.825534    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8" event={"ID":"146a6029-9c1d-4eee-8071-4482355322aa","Type":"ContainerDied","Data":"79c94cb22761da13b6913a44dfab20da8009b822e9b9ecf8447f8bbfdd177525"}
Oct 29 23:06:53.957875 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:53.957852    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:53.989709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:53.989674    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146a6029-9c1d-4eee-8071-4482355322aa-kube-api-access-gfs74\" (UniqueName: \"kubernetes.io/projected/146a6029-9c1d-4eee-8071-4482355322aa-kube-api-access-gfs74\") pod \"146a6029-9c1d-4eee-8071-4482355322aa\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") "
Oct 29 23:06:53.989709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:53.989711    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-xks-ca\") pod \"146a6029-9c1d-4eee-8071-4482355322aa\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") "
Oct 29 23:06:53.989882 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:53.989740    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-openshift-ca\") pod \"146a6029-9c1d-4eee-8071-4482355322aa\" (UID: \"146a6029-9c1d-4eee-8071-4482355322aa\") "
Oct 29 23:06:53.990117 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:53.990086    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-openshift-ca" pod "146a6029-9c1d-4eee-8071-4482355322aa" (UID: "146a6029-9c1d-4eee-8071-4482355322aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:06:53.991755 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:53.991728    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-xks-ca" pod "146a6029-9c1d-4eee-8071-4482355322aa" (UID: "146a6029-9c1d-4eee-8071-4482355322aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:06:53.991860 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:53.991830    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146a6029-9c1d-4eee-8071-4482355322aa-kube-api-access-gfs74" pod "146a6029-9c1d-4eee-8071-4482355322aa" (UID: "146a6029-9c1d-4eee-8071-4482355322aa"). InnerVolumeSpecName "kube-api-access-gfs74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:06:54.090986 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:54.090903    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:54.090986 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:54.090931    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146a6029-9c1d-4eee-8071-4482355322aa-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:54.090986 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:54.090940    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfs74\" (UniqueName: \"kubernetes.io/projected/146a6029-9c1d-4eee-8071-4482355322aa-kube-api-access-gfs74\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:06:54.837776 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:54.837747    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"
Oct 29 23:06:54.837952 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:54.837789    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8" event={"ID":"146a6029-9c1d-4eee-8071-4482355322aa","Type":"ContainerDied","Data":"793e61b9e2f67a4071e66c72e8d602f59ab70fded81840a464b3a81f074e9d97"}
Oct 29 23:06:54.837952 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:06:54.837837    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="793e61b9e2f67a4071e66c72e8d602f59ab70fded81840a464b3a81f074e9d97"
Oct 29 23:07:03.249592 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:07:03.249522    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:07:04.904727 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:04.904678    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:04.905521 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:04.905488    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:05.213230 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:05.212382    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:07:05.213230 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:05.212447    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:07:11.031426 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:11.031391    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:12.360841 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.360805    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"]
Oct 29 23:07:12.373979 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.373940    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lsxt8"]
Oct 29 23:07:12.407701 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.407672    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"]
Oct 29 23:07:12.408185 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.408166    4839 cpu_manager.go:414] "containerMap: removing container" podUID="146a6029-9c1d-4eee-8071-4482355322aa" containerName="revoke-keys"
Oct 29 23:07:12.408235 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.408190    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146a6029-9c1d-4eee-8071-4482355322aa" containerName="revoke-keys"
Oct 29 23:07:12.408334 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.408321    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="146a6029-9c1d-4eee-8071-4482355322aa" containerName="revoke-keys"
Oct 29 23:07:12.412287 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.412263    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.428006 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.427974    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"]
Oct 29 23:07:12.428344 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.428318    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:12.428451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.428328    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:07:12.428451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.428330    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:07:12.428451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.428388    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:07:12.502699 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.502666    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.502873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.502709    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6xh\" (UniqueName: \"kubernetes.io/projected/4c912e5f-98d0-45b1-b111-6b003fd4cf51-kube-api-access-rc6xh\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.502873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.502737    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.603987 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.603947    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.604195 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.604011    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6xh\" (UniqueName: \"kubernetes.io/projected/4c912e5f-98d0-45b1-b111-6b003fd4cf51-kube-api-access-rc6xh\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.604195 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.604049    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.604655 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.604632    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.606442 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.606421    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.625535 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.625470    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6xh\" (UniqueName: \"kubernetes.io/projected/4c912e5f-98d0-45b1-b111-6b003fd4cf51-kube-api-access-rc6xh\") pod \"maas-api-revoke-sub-3d4156f1-s79n5\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:12.724430 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:12.724391    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:13.008666 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:13.008587    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146a6029-9c1d-4eee-8071-4482355322aa" path="/var/lib/kubelet/pods/146a6029-9c1d-4eee-8071-4482355322aa/volumes"
Oct 29 23:07:13.019100 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:13.019069    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"]
Oct 29 23:07:13.965113 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:13.965077    4839 generic.go:356] "Generic (PLEG): container finished" podID="4c912e5f-98d0-45b1-b111-6b003fd4cf51" containerID="0575a3b641e78b10fa31b5900bc94e3ad7753110f042eea86c4ad06be7898bf3" exitCode=0
Oct 29 23:07:13.965548 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:13.965184    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5" event={"ID":"4c912e5f-98d0-45b1-b111-6b003fd4cf51","Type":"ContainerDied","Data":"0575a3b641e78b10fa31b5900bc94e3ad7753110f042eea86c4ad06be7898bf3"}
Oct 29 23:07:13.965548 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:13.965237    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5" event={"ID":"4c912e5f-98d0-45b1-b111-6b003fd4cf51","Type":"ContainerStarted","Data":"73ed167d4d5d4b9aca1da4b662f80ed7029cf80230cd42ee71b9fbeeb9cc868f"}
Oct 29 23:07:15.102197 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.102174    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:15.229375 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.229287    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-xks-ca\") pod \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") "
Oct 29 23:07:15.229528 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.229392    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-openshift-ca\") pod \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") "
Oct 29 23:07:15.229528 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.229443    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c912e5f-98d0-45b1-b111-6b003fd4cf51-kube-api-access-rc6xh\" (UniqueName: \"kubernetes.io/projected/4c912e5f-98d0-45b1-b111-6b003fd4cf51-kube-api-access-rc6xh\") pod \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\" (UID: \"4c912e5f-98d0-45b1-b111-6b003fd4cf51\") "
Oct 29 23:07:15.229801 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.229766    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-openshift-ca" pod "4c912e5f-98d0-45b1-b111-6b003fd4cf51" (UID: "4c912e5f-98d0-45b1-b111-6b003fd4cf51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:15.231483 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.231463    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-xks-ca" pod "4c912e5f-98d0-45b1-b111-6b003fd4cf51" (UID: "4c912e5f-98d0-45b1-b111-6b003fd4cf51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:15.231590 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.231529    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c912e5f-98d0-45b1-b111-6b003fd4cf51-kube-api-access-rc6xh" pod "4c912e5f-98d0-45b1-b111-6b003fd4cf51" (UID: "4c912e5f-98d0-45b1-b111-6b003fd4cf51"). InnerVolumeSpecName "kube-api-access-rc6xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:15.330281 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.330243    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6xh\" (UniqueName: \"kubernetes.io/projected/4c912e5f-98d0-45b1-b111-6b003fd4cf51-kube-api-access-rc6xh\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:07:15.330281 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.330276    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:07:15.330281 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.330285    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c912e5f-98d0-45b1-b111-6b003fd4cf51-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:07:15.976979 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.976947    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"
Oct 29 23:07:15.977192 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.976951    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5" event={"ID":"4c912e5f-98d0-45b1-b111-6b003fd4cf51","Type":"ContainerDied","Data":"73ed167d4d5d4b9aca1da4b662f80ed7029cf80230cd42ee71b9fbeeb9cc868f"}
Oct 29 23:07:15.977192 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:15.977066    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ed167d4d5d4b9aca1da4b662f80ed7029cf80230cd42ee71b9fbeeb9cc868f"
Oct 29 23:07:16.436043 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:16.436013    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:24.203531 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:24.203498    4839 scope.go:122] "RemoveContainer" containerID="602e55e23541b1dc3607f10a3f67c4f77b01825db38c74b38cd3e97cd4afcd25"
Oct 29 23:07:24.212689 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:24.212671    4839 scope.go:122] "RemoveContainer" containerID="1367aef13ca07651d6c82e1d84ef2476dedcb962acc0e3bb3323c533d120f082"
Oct 29 23:07:24.220415 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:24.220397    4839 scope.go:122] "RemoveContainer" containerID="61894711ce7df2fd16183a81833109d343b86ec2eac765c32621d198af0b3860"
Oct 29 23:07:24.227940 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:24.227919    4839 scope.go:122] "RemoveContainer" containerID="c97bb04ac83e47fc553eaf0eb92ef2302735b89cc0801ce31ec099c02caf7720"
Oct 29 23:07:24.235915 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:24.235896    4839 scope.go:122] "RemoveContainer" containerID="4be49e5a01162036fb66598818df6f4cc2cdb7630ffb926a93568f641b1fd07b"
Oct 29 23:07:33.237234 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:07:33.237192    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:07:36.046174 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.046112    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"]
Oct 29 23:07:36.059013 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.058984    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s79n5"]
Oct 29 23:07:36.095365 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.095326    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"]
Oct 29 23:07:36.095724 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.095705    4839 cpu_manager.go:414] "containerMap: removing container" podUID="4c912e5f-98d0-45b1-b111-6b003fd4cf51" containerName="revoke-keys"
Oct 29 23:07:36.095822 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.095726    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c912e5f-98d0-45b1-b111-6b003fd4cf51" containerName="revoke-keys"
Oct 29 23:07:36.095883 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.095829    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c912e5f-98d0-45b1-b111-6b003fd4cf51" containerName="revoke-keys"
Oct 29 23:07:36.100775 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.100753    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.117320 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.117285    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:07:36.117467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.117330    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:07:36.117467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.117356    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:07:36.117467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.117378    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:07:36.118256 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.118232    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"]
Oct 29 23:07:36.216963 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.216926    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqpv9\" (UniqueName: \"kubernetes.io/projected/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-kube-api-access-tqpv9\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.217190 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.216972    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.217190 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.217013    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.318234 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.318125    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqpv9\" (UniqueName: \"kubernetes.io/projected/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-kube-api-access-tqpv9\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.318234 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.318207    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.318491 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.318255    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.318935 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.318905    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.320608 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.320581    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.338476 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.338452    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqpv9\" (UniqueName: \"kubernetes.io/projected/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-kube-api-access-tqpv9\") pod \"maas-api-revoke-sub-3d4156f1-z7wsf\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.415027 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.414990    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:36.708858 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:07:36.708816    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1545e5fb_91fe_47ab_8ac6_5e000f2a1eea.slice/crio-5f32eb37fdbc354ab1c79bc0f919bad437270883efa8190ea33dd43d51bedfb5 WatchSource:0}: Error finding container 5f32eb37fdbc354ab1c79bc0f919bad437270883efa8190ea33dd43d51bedfb5: Status 404 returned error can't find the container with id 5f32eb37fdbc354ab1c79bc0f919bad437270883efa8190ea33dd43d51bedfb5
Oct 29 23:07:36.709212 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:36.709183    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"]
Oct 29 23:07:37.025249 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:37.025208    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c912e5f-98d0-45b1-b111-6b003fd4cf51" path="/var/lib/kubelet/pods/4c912e5f-98d0-45b1-b111-6b003fd4cf51/volumes"
Oct 29 23:07:37.090008 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:37.089957    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf" event={"ID":"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea","Type":"ContainerStarted","Data":"43b5a382d4615aa0bae81965c118262f3c73e40ceeb4c873750d647942153ecd"}
Oct 29 23:07:37.090415 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:37.090222    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf" event={"ID":"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea","Type":"ContainerStarted","Data":"5f32eb37fdbc354ab1c79bc0f919bad437270883efa8190ea33dd43d51bedfb5"}
Oct 29 23:07:37.133648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:37.133599    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf" podStartSLOduration=1.133584654 podStartE2EDuration="1.133584654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:07:37.131514463 +0000 UTC m=+1514.234590738" watchObservedRunningTime="2026-10-29 23:07:37.133584654 +0000 UTC m=+1514.236660925"
Oct 29 23:07:38.096999 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:38.096965    4839 generic.go:356] "Generic (PLEG): container finished" podID="1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" containerID="43b5a382d4615aa0bae81965c118262f3c73e40ceeb4c873750d647942153ecd" exitCode=0
Oct 29 23:07:38.097404 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:38.097052    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf" event={"ID":"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea","Type":"ContainerDied","Data":"43b5a382d4615aa0bae81965c118262f3c73e40ceeb4c873750d647942153ecd"}
Oct 29 23:07:39.231231 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.231206    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:39.345759 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.345725    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-xks-ca\") pod \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") "
Oct 29 23:07:39.345973 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.345783    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-kube-api-access-tqpv9\" (UniqueName: \"kubernetes.io/projected/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-kube-api-access-tqpv9\") pod \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") "
Oct 29 23:07:39.345973 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.345826    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-openshift-ca\") pod \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\" (UID: \"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea\") "
Oct 29 23:07:39.346259 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.346228    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-openshift-ca" pod "1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" (UID: "1545e5fb-91fe-47ab-8ac6-5e000f2a1eea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:07:39.347900 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.347869    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-xks-ca" pod "1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" (UID: "1545e5fb-91fe-47ab-8ac6-5e000f2a1eea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:07:39.348010 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.347952    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-kube-api-access-tqpv9" pod "1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" (UID: "1545e5fb-91fe-47ab-8ac6-5e000f2a1eea"). InnerVolumeSpecName "kube-api-access-tqpv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:07:39.446750 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.446697    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqpv9\" (UniqueName: \"kubernetes.io/projected/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-kube-api-access-tqpv9\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:07:39.446750 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.446744    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:07:39.446750 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:39.446759    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:07:40.107749 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:40.107718    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"
Oct 29 23:07:40.107927 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:40.107753    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf" event={"ID":"1545e5fb-91fe-47ab-8ac6-5e000f2a1eea","Type":"ContainerDied","Data":"5f32eb37fdbc354ab1c79bc0f919bad437270883efa8190ea33dd43d51bedfb5"}
Oct 29 23:07:40.107927 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:40.107797    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f32eb37fdbc354ab1c79bc0f919bad437270883efa8190ea33dd43d51bedfb5"
Oct 29 23:07:57.603082 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:57.603047    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:58.074634 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:58.074593    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:07:58.223568 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:07:58.223533    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:01.245335 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:01.245302    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:01.376316 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:01.376277    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:02.746213 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.746180    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"]
Oct 29 23:08:02.746700 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.746661    4839 cpu_manager.go:414] "containerMap: removing container" podUID="1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" containerName="revoke-keys"
Oct 29 23:08:02.746700 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.746680    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" containerName="revoke-keys"
Oct 29 23:08:02.746824 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.746785    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" containerName="revoke-keys"
Oct 29 23:08:02.749968 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.749946    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.767467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.767443    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"]
Oct 29 23:08:02.778784 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.778755    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:08:02.778784 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.778755    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-2b2738-serving-cert\""
Oct 29 23:08:02.778989 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.778797    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:08:02.778989 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.778830    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:08:02.781438 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.781418    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l5lq\""
Oct 29 23:08:02.833984 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.833954    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:02.850201 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.850173    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/469904d5-bdce-44a3-be10-569cf5d95f78-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.850382 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.850264    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.850382 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.850371    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.850466 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.850417    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn8bd\" (UniqueName: \"kubernetes.io/projected/469904d5-bdce-44a3-be10-569cf5d95f78-kube-api-access-qn8bd\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.935979 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.935944    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:02.950989 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.950954    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.951130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.951011    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn8bd\" (UniqueName: \"kubernetes.io/projected/469904d5-bdce-44a3-be10-569cf5d95f78-kube-api-access-qn8bd\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.951130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.951034    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/469904d5-bdce-44a3-be10-569cf5d95f78-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.951130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.951069    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.951781 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.951755    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/469904d5-bdce-44a3-be10-569cf5d95f78-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.953470 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.953440    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-metrics-certs\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.953582 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.953497    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:02.974885 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:02.974847    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn8bd\" (UniqueName: \"kubernetes.io/projected/469904d5-bdce-44a3-be10-569cf5d95f78-kube-api-access-qn8bd\") pod \"maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:03.035789 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.035752    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:03.067678 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.067648    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:03.141808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.141773    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"]
Oct 29 23:08:03.145431 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.145411    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.161990 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.161958    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"]
Oct 29 23:08:03.197240 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.197214    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-2b2738\""
Oct 29 23:08:03.242046 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.242016    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-daa70505-dockercfg-tvctm\""
Oct 29 23:08:03.246632 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:08:03.246597    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:08:03.254251 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.254228    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2489be7a-d6ee-4552-a9bf-acb3af9c0725-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.254346 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.254298    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9np6m\" (UniqueName: \"kubernetes.io/projected/2489be7a-d6ee-4552-a9bf-acb3af9c0725-kube-api-access-9np6m\") pod \"payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.355031 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.354939    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2489be7a-d6ee-4552-a9bf-acb3af9c0725-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.355235 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.355048    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9np6m\" (UniqueName: \"kubernetes.io/projected/2489be7a-d6ee-4552-a9bf-acb3af9c0725-kube-api-access-9np6m\") pod \"payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.355707 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.355683    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2489be7a-d6ee-4552-a9bf-acb3af9c0725-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.371293 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:08:03.371264    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469904d5_bdce_44a3_be10_569cf5d95f78.slice/crio-5751f9dcd5938773292499c51c48b2571e28d939665ceef7baeed8e032a5778b WatchSource:0}: Error finding container 5751f9dcd5938773292499c51c48b2571e28d939665ceef7baeed8e032a5778b: Status 404 returned error can't find the container with id 5751f9dcd5938773292499c51c48b2571e28d939665ceef7baeed8e032a5778b
Oct 29 23:08:03.371692 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.371665    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"]
Oct 29 23:08:03.372446 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.372429    4839 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:08:03.380013 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.379991    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9np6m\" (UniqueName: \"kubernetes.io/projected/2489be7a-d6ee-4552-a9bf-acb3af9c0725-kube-api-access-9np6m\") pod \"payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.468334 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.468301    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:03.761404 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:03.761374    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"]
Oct 29 23:08:03.761772 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:08:03.761628    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2489be7a_d6ee_4552_a9bf_acb3af9c0725.slice/crio-9cf53f5dc0bfac5315f3159f35463258cacb2549df5cf889517fd7ea8b675d3a WatchSource:0}: Error finding container 9cf53f5dc0bfac5315f3159f35463258cacb2549df5cf889517fd7ea8b675d3a: Status 404 returned error can't find the container with id 9cf53f5dc0bfac5315f3159f35463258cacb2549df5cf889517fd7ea8b675d3a
Oct 29 23:08:04.251113 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.251076    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng" event={"ID":"469904d5-bdce-44a3-be10-569cf5d95f78","Type":"ContainerStarted","Data":"2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681"}
Oct 29 23:08:04.251312 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.251134    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng" event={"ID":"469904d5-bdce-44a3-be10-569cf5d95f78","Type":"ContainerStarted","Data":"5751f9dcd5938773292499c51c48b2571e28d939665ceef7baeed8e032a5778b"}
Oct 29 23:08:04.251312 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.251246    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:04.252795 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.252770    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" event={"ID":"2489be7a-d6ee-4552-a9bf-acb3af9c0725","Type":"ContainerStarted","Data":"a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17"}
Oct 29 23:08:04.252920 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.252814    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" event={"ID":"2489be7a-d6ee-4552-a9bf-acb3af9c0725","Type":"ContainerStarted","Data":"9cf53f5dc0bfac5315f3159f35463258cacb2549df5cf889517fd7ea8b675d3a"}
Oct 29 23:08:04.252920 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.252855    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:04.297202 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.297133    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng" podStartSLOduration=1.838501596 podStartE2EDuration="2.297119752s" totalImagesPullingTime="458.618156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:04.294987086 +0000 UTC m=+1541.398063360" watchObservedRunningTime="2026-10-29 23:08:04.297119752 +0000 UTC m=+1541.400196030"
Oct 29 23:08:04.339210 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.339141    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" podStartSLOduration=1.339127795 podStartE2EDuration="1.339127795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:08:04.336361319 +0000 UTC m=+1541.439437589" watchObservedRunningTime="2026-10-29 23:08:04.339127795 +0000 UTC m=+1541.442204103"
Oct 29 23:08:04.516713 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:04.516677    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:06.100481 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:06.100443    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:07.909773 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:07.909738    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:09.638849 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:09.638810    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:10.264648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:10.264618    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:08:11.284276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:11.284243    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:13.167276 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:13.167247    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:14.906431 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:14.906357    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:15.262021 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:15.261936    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:08:16.157173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:16.157122    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:16.589109 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:16.589075    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:16.691173 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:16.691114    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:16.800772 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:16.800736    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:16.930696 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:16.930608    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:18.320636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:18.320602    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:18.567436 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:18.567406    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:20.029972 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:20.029931    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"]
Oct 29 23:08:20.055572 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:20.055537    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-nw5vf"]
Oct 29 23:08:20.235839 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:20.235806    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:21.007196 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:21.007166    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ab3a05-ef76-4adf-8eec-d92101fda8ee" path="/var/lib/kubelet/pods/e3ab3a05-ef76-4adf-8eec-d92101fda8ee/volumes"
Oct 29 23:08:21.962445 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:21.962400    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:23.478989 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:23.478956    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:25.182030 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:25.181993    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:26.767523 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:26.767486    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:28.500403 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:28.500371    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:30.133304 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:30.133268    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:31.828714 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:31.828676    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:33.237525 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:08:33.237491    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:08:33.468258 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:33.468219    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:35.271463 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:35.271436    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:36.840913 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:36.840879    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:38.547496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:38.547464    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:40.165733 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:40.165704    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:40.541189 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:40.541158    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:41.816827 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:41.816795    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:42.105343 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:42.105256    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:42.191477 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:42.191444    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:43.507161 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:43.507108    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:45.137900 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:45.137868    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:46.614834 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:46.614797    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:48.191317 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:48.191279    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:49.888549 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:49.888514    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:51.583928 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:51.583895    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:53.002734 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:53.002698    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:53.194532 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:53.194501    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:54.992918 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:54.992883    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:56.597863 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:56.597825    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:57.415595 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:57.415565    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:57.515978 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:57.515946    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:58.034556 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:58.034520    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:08:58.273264 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:08:58.273225    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:00.035763 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:00.035728    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:01.683297 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:01.683260    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:03.240519 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:09:03.240480    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:09:03.294077 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:03.294036    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:03.623892 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:03.623860    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"]
Oct 29 23:09:03.732645 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:03.732609    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:03.836078 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:03.836039    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:04.988991 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:04.988959    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:05.709086 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:05.709052    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:05.820756 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:05.820721    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:06.460845 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.460810    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"]
Oct 29 23:09:06.461293 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.461004    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng" podUID="469904d5-bdce-44a3-be10-569cf5d95f78" containerName="maas-api" containerID="cri-o://2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681" gracePeriod=30
Oct 29 23:09:06.645127 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.645094    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:06.937139 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.937106    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:09:06.987686 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.987656    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-maas-api-tls\") pod \"469904d5-bdce-44a3-be10-569cf5d95f78\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") "
Oct 29 23:09:06.987873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.987704    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/469904d5-bdce-44a3-be10-569cf5d95f78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/469904d5-bdce-44a3-be10-569cf5d95f78-openshift-service-ca\") pod \"469904d5-bdce-44a3-be10-569cf5d95f78\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") "
Oct 29 23:09:06.987873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.987738    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469904d5-bdce-44a3-be10-569cf5d95f78-kube-api-access-qn8bd\" (UniqueName: \"kubernetes.io/projected/469904d5-bdce-44a3-be10-569cf5d95f78-kube-api-access-qn8bd\") pod \"469904d5-bdce-44a3-be10-569cf5d95f78\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") "
Oct 29 23:09:06.987873 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.987819    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-metrics-certs\") pod \"469904d5-bdce-44a3-be10-569cf5d95f78\" (UID: \"469904d5-bdce-44a3-be10-569cf5d95f78\") "
Oct 29 23:09:06.988113 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.988089    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/469904d5-bdce-44a3-be10-569cf5d95f78-openshift-service-ca" pod "469904d5-bdce-44a3-be10-569cf5d95f78" (UID: "469904d5-bdce-44a3-be10-569cf5d95f78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:06.989985 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.989957    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469904d5-bdce-44a3-be10-569cf5d95f78-kube-api-access-qn8bd" pod "469904d5-bdce-44a3-be10-569cf5d95f78" (UID: "469904d5-bdce-44a3-be10-569cf5d95f78"). InnerVolumeSpecName "kube-api-access-qn8bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:06.990124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.989993    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-maas-api-tls" pod "469904d5-bdce-44a3-be10-569cf5d95f78" (UID: "469904d5-bdce-44a3-be10-569cf5d95f78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:06.990124 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:06.990044    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-metrics-certs" pod "469904d5-bdce-44a3-be10-569cf5d95f78" (UID: "469904d5-bdce-44a3-be10-569cf5d95f78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:07.088924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.088888    4839 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/469904d5-bdce-44a3-be10-569cf5d95f78-openshift-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:07.088924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.088920    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn8bd\" (UniqueName: \"kubernetes.io/projected/469904d5-bdce-44a3-be10-569cf5d95f78-kube-api-access-qn8bd\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:07.088924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.088930    4839 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-metrics-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:07.089199 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.088939    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/469904d5-bdce-44a3-be10-569cf5d95f78-maas-api-tls\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:07.096546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.096518    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:07.198473 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.198442    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:07.591611 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.591577    4839 generic.go:356] "Generic (PLEG): container finished" podID="469904d5-bdce-44a3-be10-569cf5d95f78" containerID="2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681" exitCode=0
Oct 29 23:09:07.592038 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.591620    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng" event={"ID":"469904d5-bdce-44a3-be10-569cf5d95f78","Type":"ContainerDied","Data":"2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681"}
Oct 29 23:09:07.592038 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.591629    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"
Oct 29 23:09:07.592038 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.591676    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng" event={"ID":"469904d5-bdce-44a3-be10-569cf5d95f78","Type":"ContainerDied","Data":"5751f9dcd5938773292499c51c48b2571e28d939665ceef7baeed8e032a5778b"}
Oct 29 23:09:07.592038 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.591706    4839 scope.go:122] "RemoveContainer" containerID="2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681"
Oct 29 23:09:07.600969 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.600950    4839 scope.go:122] "RemoveContainer" containerID="2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681"
Oct 29 23:09:07.601374 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:09:07.601351    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681\": container with ID starting with 2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681 not found: ID does not exist" containerID="2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681"
Oct 29 23:09:07.601435 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.601383    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681"} err="failed to get container status \"2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681\": rpc error: code = NotFound desc = could not find container \"2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681\": container with ID starting with 2d85cee8429bb14a8b4af9ded0ede6606b7c42862544f64b079d00c3a31d3681 not found: ID does not exist"
Oct 29 23:09:07.647995 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.647963    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"]
Oct 29 23:09:07.661914 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.661884    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-2b2738-6f7858bc45-bmhng"]
Oct 29 23:09:07.769894 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:07.769864    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:08.551451 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:08.551405    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:08.624393 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:08.624336    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" podUID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" containerName="payload-pre-processing" containerID="cri-o://a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17" gracePeriod=25
Oct 29 23:09:08.662556 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:08.662525    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:09.007312 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:09.007237    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="469904d5-bdce-44a3-be10-569cf5d95f78" path="/var/lib/kubelet/pods/469904d5-bdce-44a3-be10-569cf5d95f78/volumes"
Oct 29 23:09:09.009910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:09.009885    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:09.110497 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:09.110467    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:10.272781 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:10.272739    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:10.372415 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:10.372380    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:11.027452 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:11.027416    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"]
Oct 29 23:09:11.041045 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:11.041013    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-bgn5k"]
Oct 29 23:09:13.007829 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:13.007795    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216f56fa-6413-4a3c-b75a-ff3a201a60ab" path="/var/lib/kubelet/pods/216f56fa-6413-4a3c-b75a-ff3a201a60ab/volumes"
Oct 29 23:09:13.819502 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:13.819458    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:13.919131 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:13.919095    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:14.194376 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:14.194283    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:15.260040 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.260003    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" podUID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:09:15.510105 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.510009    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"]
Oct 29 23:09:15.526320 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.526279    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-z7wsf"]
Oct 29 23:09:15.564709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.564671    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"]
Oct 29 23:09:15.565056 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.565032    4839 cpu_manager.go:414] "containerMap: removing container" podUID="469904d5-bdce-44a3-be10-569cf5d95f78" containerName="maas-api"
Oct 29 23:09:15.565056 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.565047    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469904d5-bdce-44a3-be10-569cf5d95f78" containerName="maas-api"
Oct 29 23:09:15.565270 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.565110    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="469904d5-bdce-44a3-be10-569cf5d95f78" containerName="maas-api"
Oct 29 23:09:15.568108 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.568091    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.585023 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.584979    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:09:15.585274 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.585249    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:09:15.585431 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.585284    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:09:15.585572 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.585294    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:09:15.586074 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.586050    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"]
Oct 29 23:09:15.664542 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.664507    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.664542 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.664544    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.664752 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.664662    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpr5k\" (UniqueName: \"kubernetes.io/projected/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-kube-api-access-gpr5k\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.765666 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.765638    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpr5k\" (UniqueName: \"kubernetes.io/projected/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-kube-api-access-gpr5k\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.765830 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.765700    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.765830 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.765722    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.766381 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.766362    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.768012 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.767992    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.786518 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.786488    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpr5k\" (UniqueName: \"kubernetes.io/projected/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-kube-api-access-gpr5k\") pod \"maas-api-revoke-sub-3d4156f1-gl8j7\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:15.880293 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:15.880256    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:16.172717 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:16.172690    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"]
Oct 29 23:09:16.174286 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:09:16.174251    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a09783_fa24_4d23_bbc5_c1ff6aadbada.slice/crio-947578abc1dc79f57a7f462ae089c9061fa3beedf30f5f38c370befc6b941794 WatchSource:0}: Error finding container 947578abc1dc79f57a7f462ae089c9061fa3beedf30f5f38c370befc6b941794: Status 404 returned error can't find the container with id 947578abc1dc79f57a7f462ae089c9061fa3beedf30f5f38c370befc6b941794
Oct 29 23:09:16.640184 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:16.640127    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7" event={"ID":"c7a09783-fa24-4d23-bbc5-c1ff6aadbada","Type":"ContainerStarted","Data":"3d6dfd6aa3522935e5690fc9b74677bdc823eb94d179b9715eb7c3f794a96f3e"}
Oct 29 23:09:16.640631 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:16.640197    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7" event={"ID":"c7a09783-fa24-4d23-bbc5-c1ff6aadbada","Type":"ContainerStarted","Data":"947578abc1dc79f57a7f462ae089c9061fa3beedf30f5f38c370befc6b941794"}
Oct 29 23:09:16.681861 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:16.681804    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7" podStartSLOduration=1.6817891280000001 podStartE2EDuration="1.681789128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:09:16.681031826 +0000 UTC m=+1613.784108096" watchObservedRunningTime="2026-10-29 23:09:16.681789128 +0000 UTC m=+1613.784865446"
Oct 29 23:09:16.702542 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:16.702506    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:17.007584 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:17.007552    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1545e5fb-91fe-47ab-8ac6-5e000f2a1eea" path="/var/lib/kubelet/pods/1545e5fb-91fe-47ab-8ac6-5e000f2a1eea/volumes"
Oct 29 23:09:17.647438 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:17.647404    4839 generic.go:356] "Generic (PLEG): container finished" podID="c7a09783-fa24-4d23-bbc5-c1ff6aadbada" containerID="3d6dfd6aa3522935e5690fc9b74677bdc823eb94d179b9715eb7c3f794a96f3e" exitCode=0
Oct 29 23:09:17.647833 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:17.647488    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7" event={"ID":"c7a09783-fa24-4d23-bbc5-c1ff6aadbada","Type":"ContainerDied","Data":"3d6dfd6aa3522935e5690fc9b74677bdc823eb94d179b9715eb7c3f794a96f3e"}
Oct 29 23:09:18.782419 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.782394    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:18.893561 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.893525    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-kube-api-access-gpr5k\" (UniqueName: \"kubernetes.io/projected/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-kube-api-access-gpr5k\") pod \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") "
Oct 29 23:09:18.893737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.893584    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-openshift-ca\") pod \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") "
Oct 29 23:09:18.893737 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.893685    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-xks-ca\") pod \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\" (UID: \"c7a09783-fa24-4d23-bbc5-c1ff6aadbada\") "
Oct 29 23:09:18.894009 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.893989    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-openshift-ca" pod "c7a09783-fa24-4d23-bbc5-c1ff6aadbada" (UID: "c7a09783-fa24-4d23-bbc5-c1ff6aadbada"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:18.895740 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.895712    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-xks-ca" pod "c7a09783-fa24-4d23-bbc5-c1ff6aadbada" (UID: "c7a09783-fa24-4d23-bbc5-c1ff6aadbada"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:09:18.895814 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.895792    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-kube-api-access-gpr5k" pod "c7a09783-fa24-4d23-bbc5-c1ff6aadbada" (UID: "c7a09783-fa24-4d23-bbc5-c1ff6aadbada"). InnerVolumeSpecName "kube-api-access-gpr5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:18.994837 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.994756    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:18.994837 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.994788    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpr5k\" (UniqueName: \"kubernetes.io/projected/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-kube-api-access-gpr5k\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:18.994837 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:18.994799    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7a09783-fa24-4d23-bbc5-c1ff6aadbada-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:19.660298 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:19.660257    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7" event={"ID":"c7a09783-fa24-4d23-bbc5-c1ff6aadbada","Type":"ContainerDied","Data":"947578abc1dc79f57a7f462ae089c9061fa3beedf30f5f38c370befc6b941794"}
Oct 29 23:09:19.660298 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:19.660288    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"
Oct 29 23:09:19.660522 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:19.660319    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947578abc1dc79f57a7f462ae089c9061fa3beedf30f5f38c370befc6b941794"
Oct 29 23:09:24.281913 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:24.281880    4839 scope.go:122] "RemoveContainer" containerID="e35e065a548b8e86cc785b0119a3a6b937bbf2075c2ad79f7846b87b6f7ece86"
Oct 29 23:09:25.260097 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:25.260057    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" podUID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 29 23:09:29.519798 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.519764    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:09:29.717776 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.717676    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2489be7a-d6ee-4552-a9bf-acb3af9c0725-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2489be7a-d6ee-4552-a9bf-acb3af9c0725-plugins-config-volume\") pod \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") "
Oct 29 23:09:29.746836 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.718740    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2489be7a-d6ee-4552-a9bf-acb3af9c0725-plugins-config-volume" pod "2489be7a-d6ee-4552-a9bf-acb3af9c0725" (UID: "2489be7a-d6ee-4552-a9bf-acb3af9c0725"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:09:29.747804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.747281    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2489be7a-d6ee-4552-a9bf-acb3af9c0725-kube-api-access-9np6m\" (UniqueName: \"kubernetes.io/projected/2489be7a-d6ee-4552-a9bf-acb3af9c0725-kube-api-access-9np6m\") pod \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\" (UID: \"2489be7a-d6ee-4552-a9bf-acb3af9c0725\") "
Oct 29 23:09:29.748350 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.748328    4839 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2489be7a-d6ee-4552-a9bf-acb3af9c0725-plugins-config-volume\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.768572 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.768318    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2489be7a-d6ee-4552-a9bf-acb3af9c0725-kube-api-access-9np6m" pod "2489be7a-d6ee-4552-a9bf-acb3af9c0725" (UID: "2489be7a-d6ee-4552-a9bf-acb3af9c0725"). InnerVolumeSpecName "kube-api-access-9np6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:09:29.790192 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.781701    4839 generic.go:356] "Generic (PLEG): container finished" podID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" containerID="a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17" exitCode=0
Oct 29 23:09:29.790192 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.781779    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" event={"ID":"2489be7a-d6ee-4552-a9bf-acb3af9c0725","Type":"ContainerDied","Data":"a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17"}
Oct 29 23:09:29.790192 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.781833    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f" event={"ID":"2489be7a-d6ee-4552-a9bf-acb3af9c0725","Type":"ContainerDied","Data":"9cf53f5dc0bfac5315f3159f35463258cacb2549df5cf889517fd7ea8b675d3a"}
Oct 29 23:09:29.790192 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.781864    4839 scope.go:122] "RemoveContainer" containerID="a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17"
Oct 29 23:09:29.790192 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.782044    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"
Oct 29 23:09:29.850510 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.850481    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9np6m\" (UniqueName: \"kubernetes.io/projected/2489be7a-d6ee-4552-a9bf-acb3af9c0725-kube-api-access-9np6m\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:09:29.855716 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.855665    4839 scope.go:122] "RemoveContainer" containerID="a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17"
Oct 29 23:09:29.856228 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:09:29.856124    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17\": container with ID starting with a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17 not found: ID does not exist" containerID="a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17"
Oct 29 23:09:29.856228 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.856183    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17"} err="failed to get container status \"a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17\": rpc error: code = NotFound desc = could not find container \"a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17\": container with ID starting with a8e170b00a84adb6660b73376132c10cfe3e462ac3f753530260312e119b4b17 not found: ID does not exist"
Oct 29 23:09:29.868079 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.868031    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"]
Oct 29 23:09:29.888694 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:29.888655    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-2b2738-b4d75j6j7f"]
Oct 29 23:09:31.006734 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:31.006695    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" path="/var/lib/kubelet/pods/2489be7a-d6ee-4552-a9bf-acb3af9c0725/volumes"
Oct 29 23:09:31.026925 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:31.026895    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"]
Oct 29 23:09:31.040551 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:31.040518    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-qdk76"]
Oct 29 23:09:33.007095 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:33.007054    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc63429-4232-4f69-98d8-e34fb6c2667c" path="/var/lib/kubelet/pods/4fc63429-4232-4f69-98d8-e34fb6c2667c/volumes"
Oct 29 23:09:33.237057 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:09:33.237021    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:09:34.393939 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:34.393901    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:35.295109 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:35.295072    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:40.950916 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:40.950886    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:09:42.024307 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:42.024274    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"]
Oct 29 23:09:42.037636 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:42.037606    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-74g96"]
Oct 29 23:09:43.006879 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:09:43.006844    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="584b3458-109e-4f7a-a384-4aa0fd7799fa" path="/var/lib/kubelet/pods/584b3458-109e-4f7a-a384-4aa0fd7799fa/volumes"
Oct 29 23:10:03.236759 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:10:03.236718    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:10:10.038015 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:10.037980    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"]
Oct 29 23:10:10.050474 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:10.050441    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-hbjtd"]
Oct 29 23:10:11.006346 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:11.006309    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7" path="/var/lib/kubelet/pods/d0f2e3f5-8cb3-4d17-a99f-9edaf593baa7/volumes"
Oct 29 23:10:16.810316 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:16.810278    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:21.029532 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:21.029491    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"]
Oct 29 23:10:21.034260 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:21.034233    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-q6wwk"]
Oct 29 23:10:23.007673 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:23.007639    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f2d491-99f5-4220-90a3-7ba792808575" path="/var/lib/kubelet/pods/07f2d491-99f5-4220-90a3-7ba792808575/volumes"
Oct 29 23:10:24.310564 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:24.310531    4839 scope.go:122] "RemoveContainer" containerID="a342393d88c44b3eefc06d9086e1fb91375eae7e313ffcdec4459d77e602dab2"
Oct 29 23:10:33.237124 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:10:33.237086    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:10:42.026163 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:42.026120    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"]
Oct 29 23:10:42.038678 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:42.038647    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7l9dj"]
Oct 29 23:10:43.007033 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:43.006996    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b942b226-bc27-446f-bffc-8fc8aed9acff" path="/var/lib/kubelet/pods/b942b226-bc27-446f-bffc-8fc8aed9acff/volumes"
Oct 29 23:10:55.671721 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:55.671686    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:10:55.772322 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:10:55.772281    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:03.306597 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:11:03.306540    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:11:04.030398 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:04.030356    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"]
Oct 29 23:11:04.043889 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:04.043858    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-t5l2x"]
Oct 29 23:11:05.006971 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.006937    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac92e5cd-f044-4c43-b04e-243220fc1474" path="/var/lib/kubelet/pods/ac92e5cd-f044-4c43-b04e-243220fc1474/volumes"
Oct 29 23:11:05.352747 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.352709    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"]
Oct 29 23:11:05.353047 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.353034    4839 cpu_manager.go:414] "containerMap: removing container" podUID="c7a09783-fa24-4d23-bbc5-c1ff6aadbada" containerName="revoke-keys"
Oct 29 23:11:05.353104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.353048    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a09783-fa24-4d23-bbc5-c1ff6aadbada" containerName="revoke-keys"
Oct 29 23:11:05.353104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.353073    4839 cpu_manager.go:414] "containerMap: removing container" podUID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" containerName="payload-pre-processing"
Oct 29 23:11:05.353104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.353078    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" containerName="payload-pre-processing"
Oct 29 23:11:05.353243 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.353131    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a09783-fa24-4d23-bbc5-c1ff6aadbada" containerName="revoke-keys"
Oct 29 23:11:05.353243 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.353167    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="2489be7a-d6ee-4552-a9bf-acb3af9c0725" containerName="payload-pre-processing"
Oct 29 23:11:05.356482 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.356466    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.373604 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.373575    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"]
Oct 29 23:11:05.383670 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.383642    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:11:05.383670 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.383671    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:11:05.383874 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.383849    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:05.383915 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.383889    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:11:05.440598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.440565    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.440789 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.440685    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.440789 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.440760    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7mh9\" (UniqueName: \"kubernetes.io/projected/8289e483-ab28-425a-be97-a0b67295bbf6-kube-api-access-q7mh9\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.541935 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.541893    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mh9\" (UniqueName: \"kubernetes.io/projected/8289e483-ab28-425a-be97-a0b67295bbf6-kube-api-access-q7mh9\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.541935 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.541941    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.542195 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.541996    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.542628 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.542606    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.544404 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.544383    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.561354 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.561325    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mh9\" (UniqueName: \"kubernetes.io/projected/8289e483-ab28-425a-be97-a0b67295bbf6-kube-api-access-q7mh9\") pod \"maas-api-revoke-sub-d576e68b-cwkfv\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.668792 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.668709    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:05.960555 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:05.960524    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"]
Oct 29 23:11:05.960896 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:11:05.960874    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8289e483_ab28_425a_be97_a0b67295bbf6.slice/crio-d1b7e676e4fa2df517f8b772c8f1a99ac22cfe0a0832192012375c0d89339116 WatchSource:0}: Error finding container d1b7e676e4fa2df517f8b772c8f1a99ac22cfe0a0832192012375c0d89339116: Status 404 returned error can't find the container with id d1b7e676e4fa2df517f8b772c8f1a99ac22cfe0a0832192012375c0d89339116
Oct 29 23:11:06.286598 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:06.286566    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv" event={"ID":"8289e483-ab28-425a-be97-a0b67295bbf6","Type":"ContainerStarted","Data":"8e796ba177bd2fecb68ba1e210b2dddf85851bc8340bf83b7d28af5d57a3cbfe"}
Oct 29 23:11:06.286939 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:06.286620    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv" event={"ID":"8289e483-ab28-425a-be97-a0b67295bbf6","Type":"ContainerStarted","Data":"d1b7e676e4fa2df517f8b772c8f1a99ac22cfe0a0832192012375c0d89339116"}
Oct 29 23:11:06.325574 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:06.325524    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv" podStartSLOduration=1.325509558 podStartE2EDuration="1.325509558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:06.32459472 +0000 UTC m=+1723.427670993" watchObservedRunningTime="2026-10-29 23:11:06.325509558 +0000 UTC m=+1723.428585867"
Oct 29 23:11:07.295729 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:07.295692    4839 generic.go:356] "Generic (PLEG): container finished" podID="8289e483-ab28-425a-be97-a0b67295bbf6" containerID="8e796ba177bd2fecb68ba1e210b2dddf85851bc8340bf83b7d28af5d57a3cbfe" exitCode=0
Oct 29 23:11:07.296116 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:07.295786    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv" event={"ID":"8289e483-ab28-425a-be97-a0b67295bbf6","Type":"ContainerDied","Data":"8e796ba177bd2fecb68ba1e210b2dddf85851bc8340bf83b7d28af5d57a3cbfe"}
Oct 29 23:11:08.429529 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.429504    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:08.569650 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.569556    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-xks-ca\") pod \"8289e483-ab28-425a-be97-a0b67295bbf6\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") "
Oct 29 23:11:08.569823 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.569663    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-openshift-ca\") pod \"8289e483-ab28-425a-be97-a0b67295bbf6\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") "
Oct 29 23:11:08.569823 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.569700    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8289e483-ab28-425a-be97-a0b67295bbf6-kube-api-access-q7mh9\" (UniqueName: \"kubernetes.io/projected/8289e483-ab28-425a-be97-a0b67295bbf6-kube-api-access-q7mh9\") pod \"8289e483-ab28-425a-be97-a0b67295bbf6\" (UID: \"8289e483-ab28-425a-be97-a0b67295bbf6\") "
Oct 29 23:11:08.570182 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.570116    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-openshift-ca" pod "8289e483-ab28-425a-be97-a0b67295bbf6" (UID: "8289e483-ab28-425a-be97-a0b67295bbf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:08.571612 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.571587    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-xks-ca" pod "8289e483-ab28-425a-be97-a0b67295bbf6" (UID: "8289e483-ab28-425a-be97-a0b67295bbf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:08.571797 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.571775    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8289e483-ab28-425a-be97-a0b67295bbf6-kube-api-access-q7mh9" pod "8289e483-ab28-425a-be97-a0b67295bbf6" (UID: "8289e483-ab28-425a-be97-a0b67295bbf6"). InnerVolumeSpecName "kube-api-access-q7mh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:08.670905 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.670873    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:11:08.670905 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.670899    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8289e483-ab28-425a-be97-a0b67295bbf6-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:11:08.670905 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:08.670909    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7mh9\" (UniqueName: \"kubernetes.io/projected/8289e483-ab28-425a-be97-a0b67295bbf6-kube-api-access-q7mh9\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:11:09.308528 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:09.308481    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"
Oct 29 23:11:09.308528 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:09.308496    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv" event={"ID":"8289e483-ab28-425a-be97-a0b67295bbf6","Type":"ContainerDied","Data":"d1b7e676e4fa2df517f8b772c8f1a99ac22cfe0a0832192012375c0d89339116"}
Oct 29 23:11:09.308804 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:09.308551    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b7e676e4fa2df517f8b772c8f1a99ac22cfe0a0832192012375c0d89339116"
Oct 29 23:11:16.931461 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:16.931423    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:20.506833 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.506765    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"]
Oct 29 23:11:20.507648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.507288    4839 cpu_manager.go:414] "containerMap: removing container" podUID="8289e483-ab28-425a-be97-a0b67295bbf6" containerName="revoke-keys"
Oct 29 23:11:20.507648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.507310    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8289e483-ab28-425a-be97-a0b67295bbf6" containerName="revoke-keys"
Oct 29 23:11:20.507648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.507414    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="8289e483-ab28-425a-be97-a0b67295bbf6" containerName="revoke-keys"
Oct 29 23:11:20.602172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.592379    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"]
Oct 29 23:11:20.602172 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.593424    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.664791 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.654037    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:11:20.664791 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.654084    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:11:20.664791 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.654134    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:11:20.664791 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.654275    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:11:20.814713 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.814664    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.829337 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.829299    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.829672 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.829652    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsl4m\" (UniqueName: \"kubernetes.io/projected/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-kube-api-access-fsl4m\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.931667 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.931621    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.931954 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.931927    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.932136 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.932117    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsl4m\" (UniqueName: \"kubernetes.io/projected/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-kube-api-access-fsl4m\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.937171 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.933472    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.955820 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.955789    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:20.957572 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:20.957508    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsl4m\" (UniqueName: \"kubernetes.io/projected/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-kube-api-access-fsl4m\") pod \"maas-api-revoke-sub-d2bc93bc-hxcdj\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:21.092913 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:21.092821    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:21.856652 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:21.856555    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"]
Oct 29 23:11:22.514935 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:22.514847    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj" event={"ID":"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e","Type":"ContainerStarted","Data":"9e5c8f71de6ad986f42c52a0be3f68e55e1f3a3e4e332ca2629a3e7bb6ee0ac9"}
Oct 29 23:11:22.514935 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:22.514907    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj" event={"ID":"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e","Type":"ContainerStarted","Data":"e5096c7843bc5ad0baab1a01dc433c41ece9850371f8eece174dc171405228a9"}
Oct 29 23:11:22.554837 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:22.554784    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj" podStartSLOduration=2.554769191 podStartE2EDuration="2.554769191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:22.553270001 +0000 UTC m=+1739.656346271" watchObservedRunningTime="2026-10-29 23:11:22.554769191 +0000 UTC m=+1739.657845462"
Oct 29 23:11:23.031347 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:23.031315    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"]
Oct 29 23:11:23.043959 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:23.043928    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-smwsk"]
Oct 29 23:11:23.521578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:23.521538    4839 generic.go:356] "Generic (PLEG): container finished" podID="a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" containerID="9e5c8f71de6ad986f42c52a0be3f68e55e1f3a3e4e332ca2629a3e7bb6ee0ac9" exitCode=0
Oct 29 23:11:23.521741 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:23.521595    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj" event={"ID":"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e","Type":"ContainerDied","Data":"9e5c8f71de6ad986f42c52a0be3f68e55e1f3a3e4e332ca2629a3e7bb6ee0ac9"}
Oct 29 23:11:24.336919 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.336889    4839 scope.go:122] "RemoveContainer" containerID="b73a6967b31df59c458c4a2d9ab61c7c9ce48901ef6c903d2543892559c3a3c2"
Oct 29 23:11:24.345369 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.345344    4839 scope.go:122] "RemoveContainer" containerID="45432271ad4edaa6fd2468cf0f4ed9feb01aafc7367ad3fd62c4af7d2d974ab1"
Oct 29 23:11:24.353167 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.353131    4839 scope.go:122] "RemoveContainer" containerID="500ed6f4eca398047716203fa2f7f48b01c79a9682ea95c876ed562d6370d77a"
Oct 29 23:11:24.360996 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.360974    4839 scope.go:122] "RemoveContainer" containerID="d68da4befb5514cc1463dd4dce9d883a348fcefb996eefe2bf34b9edea4e0fc8"
Oct 29 23:11:24.648716 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.648690    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:24.704925 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.704890    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-openshift-ca\") pod \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") "
Oct 29 23:11:24.705115 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.704946    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-xks-ca\") pod \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") "
Oct 29 23:11:24.705115 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.705071    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-kube-api-access-fsl4m\" (UniqueName: \"kubernetes.io/projected/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-kube-api-access-fsl4m\") pod \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\" (UID: \"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e\") "
Oct 29 23:11:24.705342 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.705318    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-openshift-ca" pod "a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" (UID: "a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:11:24.707061 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.707028    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-xks-ca" pod "a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" (UID: "a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:11:24.707221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.707122    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-kube-api-access-fsl4m" pod "a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" (UID: "a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e"). InnerVolumeSpecName "kube-api-access-fsl4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:11:24.806444 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.806406    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsl4m\" (UniqueName: \"kubernetes.io/projected/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-kube-api-access-fsl4m\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:11:24.806444 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.806436    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:11:24.806444 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:24.806446    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:11:25.007613 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:25.007543    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="563a1648-8fd8-45b8-a852-a07a4101dc17" path="/var/lib/kubelet/pods/563a1648-8fd8-45b8-a852-a07a4101dc17/volumes"
Oct 29 23:11:25.533141 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:25.533110    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"
Oct 29 23:11:25.533141 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:25.533157    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj" event={"ID":"a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e","Type":"ContainerDied","Data":"e5096c7843bc5ad0baab1a01dc433c41ece9850371f8eece174dc171405228a9"}
Oct 29 23:11:25.533627 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:25.533206    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5096c7843bc5ad0baab1a01dc433c41ece9850371f8eece174dc171405228a9"
Oct 29 23:11:32.197383 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:32.197349    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:32.687872 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:32.687834    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:32.831886 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:32.831852    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:33.237222 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:11:33.237190    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:11:36.281880 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:36.281797    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:36.388253 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:36.388221    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:37.969589 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:37.969557    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:38.072405 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.072370    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:38.094987 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.094952    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"]
Oct 29 23:11:38.095340 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.095324    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" containerName="revoke-keys"
Oct 29 23:11:38.095439 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.095341    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" containerName="revoke-keys"
Oct 29 23:11:38.095439 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.095427    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" containerName="revoke-keys"
Oct 29 23:11:38.099616 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.099593    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.117802 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.117774    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"]
Oct 29 23:11:38.119643 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.119610    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkd6\" (UniqueName: \"kubernetes.io/projected/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-kube-api-access-glkd6\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.119824 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.119675    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.119889 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.119813    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.119950 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.119888    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.128728 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.128689    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:11:38.129286 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.129263    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:11:38.183469 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.183438    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:38.197359 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.197331    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-a545e2-serving-cert\""
Oct 29 23:11:38.197600 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.197577    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l5lq\""
Oct 29 23:11:38.197656 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.197582    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:11:38.221376 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.221294    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.221376 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.221363    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkd6\" (UniqueName: \"kubernetes.io/projected/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-kube-api-access-glkd6\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.221599 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.221410    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.221599 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.221480    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.221719 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:11:38.221614    4839 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-serving-cert: secret "maas-api-e2e-worker-single-main-a545e2-serving-cert" not found
Oct 29 23:11:38.221719 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:11:38.221696    4839 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls podName:0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76 nodeName:}" failed. No retries permitted until 2026-10-29 23:11:38.721674324 +0000 UTC m=+1755.824750596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls") pod "maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4" (UID: "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76") : secret "maas-api-e2e-worker-single-main-a545e2-serving-cert" not found
Oct 29 23:11:38.222074 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.222052    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.223843 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.223817    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-metrics-certs\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.241952 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.241919    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkd6\" (UniqueName: \"kubernetes.io/projected/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-kube-api-access-glkd6\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.291355 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.291311    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:38.370630 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.370592    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:38.725648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.725609    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:38.728075 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:38.728050    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:39.013233 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:39.013158    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:39.304842 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:39.304803    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"]
Oct 29 23:11:39.305422 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:11:39.305389    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7ce8a2_d1f4_4452_a429_ea17fb0b1f76.slice/crio-7b34fd571ec88f1b1e67c1c135533b9ee339db235b5da5338f9378f9b15ad661 WatchSource:0}: Error finding container 7b34fd571ec88f1b1e67c1c135533b9ee339db235b5da5338f9378f9b15ad661: Status 404 returned error can't find the container with id 7b34fd571ec88f1b1e67c1c135533b9ee339db235b5da5338f9378f9b15ad661
Oct 29 23:11:39.603204 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:39.603084    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4" event={"ID":"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76","Type":"ContainerStarted","Data":"7b34fd571ec88f1b1e67c1c135533b9ee339db235b5da5338f9378f9b15ad661"}
Oct 29 23:11:40.018806 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:40.018757    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:40.610118 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:40.610083    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4" event={"ID":"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76","Type":"ContainerStarted","Data":"fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302"}
Oct 29 23:11:40.610315 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:40.610189    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:41.879907 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:41.879873    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:43.025478 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:43.025424    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4" podStartSLOduration=4.605653033 podStartE2EDuration="5.025407135s" totalImagesPullingTime="419.754102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:11:40.649261407 +0000 UTC m=+1757.752337675" watchObservedRunningTime="2026-10-29 23:11:43.025407135 +0000 UTC m=+1760.128483405"
Oct 29 23:11:43.026446 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:43.026418    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"]
Oct 29 23:11:43.038553 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:43.038526    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-ptmkw"]
Oct 29 23:11:43.651671 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:43.651633    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:45.007843 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:45.007807    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351" path="/var/lib/kubelet/pods/2917e9a9-0e6d-4b37-b8c2-7f6ac98e6351/volumes"
Oct 29 23:11:45.477736 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:45.477704    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:46.620531 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:46.620503    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:11:47.301748 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:47.301694    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:49.161542 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:49.161508    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:50.916898 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:50.916864    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:51.154221 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:51.154183    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:51.779679 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:51.779647    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:51.880054 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:51.879968    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:51.980397 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:51.980359    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:52.693835 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:52.693801    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:54.398170 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:54.398110    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:54.493087 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:54.493050    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:56.366646 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:56.366616    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:57.658966 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:57.658933    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:58.120496 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:58.120462    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:11:59.845206 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:11:59.845170    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:01.536369 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:01.536338    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:03.237579 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:12:03.237541    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:12:03.424114 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:03.424077    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:03.522643 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:03.522612    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:05.200563 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:05.200530    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:06.979758 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:06.979719    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:08.631113 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:08.631081    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:10.362072 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:10.362041    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:12.194759 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:12.194724    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:13.852118 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:13.852078    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:15.599302 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:15.599268    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:17.052977 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:17.052946    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:17.150858 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:17.150825    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:17.344120 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:17.344039    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:19.170162 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:19.170105    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:20.819035 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:20.818999    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:22.455654 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:22.455626    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:23.714282 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:23.714249    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:24.308460 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:24.308429    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:24.394263 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:24.394231    4839 scope.go:122] "RemoveContainer" containerID="d49d4069c886765a20bdec4e9a3e8cf7ce44b54471f5fd1d4e428b7de0d9236e"
Oct 29 23:12:24.402898 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:24.402873    4839 scope.go:122] "RemoveContainer" containerID="c42d44835c750c93feb26972824e08063995dbea74ba8b9ca90a440b55676cdb"
Oct 29 23:12:24.410851 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:24.410829    4839 scope.go:122] "RemoveContainer" containerID="0ca764c02430ac96ddc089724b417e76ae151efb395db10e7525c4aed5d43647"
Oct 29 23:12:26.104255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:26.104216    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:27.962433 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:27.962395    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:28.748436 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:28.748400    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:29.751901 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:29.748184    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:31.440244 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:31.440211    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:33.143924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:33.143892    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:33.238082 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:12:33.238049    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:12:34.569727 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:34.569696    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:34.614427 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:34.614393    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"]
Oct 29 23:12:34.614910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:34.614865    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4" podUID="0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" containerName="maas-api" containerID="cri-o://fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302" gracePeriod=30
Oct 29 23:12:34.660626 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:34.660591    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:34.764098 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:34.764066    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:35.850057 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.850027    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:35.863119 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.863095    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:12:35.893504 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.893476    4839 generic.go:356] "Generic (PLEG): container finished" podID="0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" containerID="fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302" exitCode=0
Oct 29 23:12:35.893659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.893563    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"
Oct 29 23:12:35.893659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.893565    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4" event={"ID":"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76","Type":"ContainerDied","Data":"fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302"}
Oct 29 23:12:35.893659 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.893635    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4" event={"ID":"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76","Type":"ContainerDied","Data":"7b34fd571ec88f1b1e67c1c135533b9ee339db235b5da5338f9378f9b15ad661"}
Oct 29 23:12:35.893784 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.893672    4839 scope.go:122] "RemoveContainer" containerID="fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302"
Oct 29 23:12:35.902948 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.902922    4839 scope.go:122] "RemoveContainer" containerID="fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302"
Oct 29 23:12:35.903276 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:12:35.903251    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302\": container with ID starting with fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302 not found: ID does not exist" containerID="fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302"
Oct 29 23:12:35.903379 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.903286    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302"} err="failed to get container status \"fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302\": rpc error: code = NotFound desc = could not find container \"fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302\": container with ID starting with fc8fb797e8d218b3eaddff80b3dd4f03e60b21b4bf616af150f078be8c722302 not found: ID does not exist"
Oct 29 23:12:35.950874 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.950796    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls\") pod \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") "
Oct 29 23:12:35.950874 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.950849    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-kube-api-access-glkd6\" (UniqueName: \"kubernetes.io/projected/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-kube-api-access-glkd6\") pod \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") "
Oct 29 23:12:35.951104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.950882    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-openshift-service-ca\") pod \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") "
Oct 29 23:12:35.951104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.950978    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-metrics-certs\") pod \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\" (UID: \"0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76\") "
Oct 29 23:12:35.951261 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.951196    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-openshift-service-ca" pod "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" (UID: "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:12:35.951398 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.951379    4839 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-openshift-service-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:12:35.953053 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.953020    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-kube-api-access-glkd6" pod "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" (UID: "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76"). InnerVolumeSpecName "kube-api-access-glkd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:12:35.953266 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.953075    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls" pod "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" (UID: "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:35.953266 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.953109    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-metrics-certs" pod "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" (UID: "0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:12:35.954763 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:35.954735    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:36.052583 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:36.052545    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkd6\" (UniqueName: \"kubernetes.io/projected/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-kube-api-access-glkd6\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:12:36.052583 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:36.052577    4839 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-metrics-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:12:36.052583 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:36.052589    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76-maas-api-tls\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:12:36.238544 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:36.238499    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"]
Oct 29 23:12:36.251042 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:36.251000    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-a545e2-68c896d5f5-xzpz4"]
Oct 29 23:12:36.484546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:36.484460    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:37.007543 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:37.007505    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" path="/var/lib/kubelet/pods/0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76/volumes"
Oct 29 23:12:42.641639 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:42.641603    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:42.743903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:42.743864    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:43.035367 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:43.035333    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:47.328066 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.328033    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"]
Oct 29 23:12:47.328576 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.328403    4839 cpu_manager.go:414] "containerMap: removing container" podUID="0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" containerName="maas-api"
Oct 29 23:12:47.328576 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.328417    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" containerName="maas-api"
Oct 29 23:12:47.328576 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.328475    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a7ce8a2-d1f4-4452-a429-ea17fb0b1f76" containerName="maas-api"
Oct 29 23:12:47.331585 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.331562    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.354463 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.354432    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd769a0-3d98-46c5-801a-6fe2328376c6-tls-certs\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.354628 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.354506    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766r5\" (UniqueName: \"kubernetes.io/projected/2bd769a0-3d98-46c5-801a-6fe2328376c6-kube-api-access-766r5\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.354628 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.354587    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-model-cache\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.354748 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.354659    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-tmp-dir\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.354748 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.354699    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-dshm\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.354874 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.354761    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-home\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.361643 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.361616    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-95f34a42-kserve-self-signed-certs\""
Oct 29 23:12:47.363441 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.363412    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"]
Oct 29 23:12:47.395394 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.395368    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:12:47.455547 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455510    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-home\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.455738 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455559    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd769a0-3d98-46c5-801a-6fe2328376c6-tls-certs\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.455738 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455599    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766r5\" (UniqueName: \"kubernetes.io/projected/2bd769a0-3d98-46c5-801a-6fe2328376c6-kube-api-access-766r5\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.455738 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455616    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-model-cache\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.455738 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455642    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-tmp-dir\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.455738 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455662    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-dshm\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.456015 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455965    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-home\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.456015 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.455978    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-tmp-dir\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.456114 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.456043    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-model-cache\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.457916 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.457894    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-dshm\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.458217 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.458200    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd769a0-3d98-46c5-801a-6fe2328376c6-tls-certs\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.476477 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.476445    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766r5\" (UniqueName: \"kubernetes.io/projected/2bd769a0-3d98-46c5-801a-6fe2328376c6-kube-api-access-766r5\") pod \"e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") " pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.647334 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.647235    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:47.943663 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.943629    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"]
Oct 29 23:12:47.943797 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:12:47.943721    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bd769a0_3d98_46c5_801a_6fe2328376c6.slice/crio-7dcc02b53ebf78250a5a9cf587e2c1b525c501b1994457b77eda3a7b82e67ff7 WatchSource:0}: Error finding container 7dcc02b53ebf78250a5a9cf587e2c1b525c501b1994457b77eda3a7b82e67ff7: Status 404 returned error can't find the container with id 7dcc02b53ebf78250a5a9cf587e2c1b525c501b1994457b77eda3a7b82e67ff7
Oct 29 23:12:47.956850 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:47.956820    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79" event={"ID":"2bd769a0-3d98-46c5-801a-6fe2328376c6","Type":"ContainerStarted","Data":"7dcc02b53ebf78250a5a9cf587e2c1b525c501b1994457b77eda3a7b82e67ff7"}
Oct 29 23:12:48.963022 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:48.962981    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79" event={"ID":"2bd769a0-3d98-46c5-801a-6fe2328376c6","Type":"ContainerStarted","Data":"647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae"}
Oct 29 23:12:49.014351 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:49.014302    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79" podStartSLOduration=2.014286453 podStartE2EDuration="2.014286453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:12:49.013270767 +0000 UTC m=+1826.116347030" watchObservedRunningTime="2026-10-29 23:12:49.014286453 +0000 UTC m=+1826.117362724"
Oct 29 23:12:49.648217 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:49.648183    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:57.647917 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:57.647869    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:57.660324 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:57.660300    4839 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:12:58.033584 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:12:58.033551    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:13:03.236796 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:13:03.236761    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:13:04.335558 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:04.335524    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:04.879195 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:04.879156    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:06.436745 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:06.436662    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:12.092937 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:12.092901    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:17.235527 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:17.235493    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:23.016360 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.016321    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"]
Oct 29 23:13:23.022285 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.022254    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.050050 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.050019    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"]
Oct 29 23:13:23.054101 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.054076    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:13:23.056417 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.054524    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:13:23.056578 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.054628    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:13:23.056692 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.055269    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:13:23.151826 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.151784    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.152052 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.151880    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpmv\" (UniqueName: \"kubernetes.io/projected/d86bcb95-5859-4b59-bc5c-030ab5061fbf-kube-api-access-mdpmv\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.152052 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.151963    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.253047 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.252993    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpmv\" (UniqueName: \"kubernetes.io/projected/d86bcb95-5859-4b59-bc5c-030ab5061fbf-kube-api-access-mdpmv\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.253313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.253055    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.253313 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.253195    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.254099 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.254065    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.256540 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.256508    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.275749 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.275674    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpmv\" (UniqueName: \"kubernetes.io/projected/d86bcb95-5859-4b59-bc5c-030ab5061fbf-kube-api-access-mdpmv\") pod \"maas-api-revoke-sub-452b6d7b-h4g78\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.361832 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.361789    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:23.688218 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:23.682546    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"]
Oct 29 23:13:23.705495 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:13:23.702831    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd86bcb95_5859_4b59_bc5c_030ab5061fbf.slice/crio-eae318cde91df035e2a5ee2ee9cb1e433918f84a404b3bedfca8aefd3601156a WatchSource:0}: Error finding container eae318cde91df035e2a5ee2ee9cb1e433918f84a404b3bedfca8aefd3601156a: Status 404 returned error can't find the container with id eae318cde91df035e2a5ee2ee9cb1e433918f84a404b3bedfca8aefd3601156a
Oct 29 23:13:24.187489 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:24.187447    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78" event={"ID":"d86bcb95-5859-4b59-bc5c-030ab5061fbf","Type":"ContainerStarted","Data":"0db7a5db08b4d5f6b0846bb8916d4d40aa0dc75cd0c86972edd1a9d81da2e614"}
Oct 29 23:13:24.193045 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:24.192237    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78" event={"ID":"d86bcb95-5859-4b59-bc5c-030ab5061fbf","Type":"ContainerStarted","Data":"eae318cde91df035e2a5ee2ee9cb1e433918f84a404b3bedfca8aefd3601156a"}
Oct 29 23:13:24.440228 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:24.438613    4839 scope.go:122] "RemoveContainer" containerID="0575a3b641e78b10fa31b5900bc94e3ad7753110f042eea86c4ad06be7898bf3"
Oct 29 23:13:24.471354 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:24.470573    4839 scope.go:122] "RemoveContainer" containerID="79c94cb22761da13b6913a44dfab20da8009b822e9b9ecf8447f8bbfdd177525"
Oct 29 23:13:24.528326 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:24.527935    4839 scope.go:122] "RemoveContainer" containerID="62c540a16368adef924ac9ea70f42dbc5bbefa159be2a4f5eb2ff2b7461f5270"
Oct 29 23:13:25.210231 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:25.207302    4839 generic.go:356] "Generic (PLEG): container finished" podID="d86bcb95-5859-4b59-bc5c-030ab5061fbf" containerID="0db7a5db08b4d5f6b0846bb8916d4d40aa0dc75cd0c86972edd1a9d81da2e614" exitCode=0
Oct 29 23:13:25.210231 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:25.207425    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78" event={"ID":"d86bcb95-5859-4b59-bc5c-030ab5061fbf","Type":"ContainerDied","Data":"0db7a5db08b4d5f6b0846bb8916d4d40aa0dc75cd0c86972edd1a9d81da2e614"}
Oct 29 23:13:26.341859 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.341833    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:26.352960 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.352937    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-xks-ca\") pod \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") "
Oct 29 23:13:26.353057 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.352980    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-openshift-ca\") pod \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") "
Oct 29 23:13:26.353120 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.353102    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86bcb95-5859-4b59-bc5c-030ab5061fbf-kube-api-access-mdpmv\" (UniqueName: \"kubernetes.io/projected/d86bcb95-5859-4b59-bc5c-030ab5061fbf-kube-api-access-mdpmv\") pod \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\" (UID: \"d86bcb95-5859-4b59-bc5c-030ab5061fbf\") "
Oct 29 23:13:26.353368 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.353345    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-openshift-ca" pod "d86bcb95-5859-4b59-bc5c-030ab5061fbf" (UID: "d86bcb95-5859-4b59-bc5c-030ab5061fbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:13:26.355076 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.355047    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-xks-ca" pod "d86bcb95-5859-4b59-bc5c-030ab5061fbf" (UID: "d86bcb95-5859-4b59-bc5c-030ab5061fbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:13:26.355202 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.355107    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86bcb95-5859-4b59-bc5c-030ab5061fbf-kube-api-access-mdpmv" pod "d86bcb95-5859-4b59-bc5c-030ab5061fbf" (UID: "d86bcb95-5859-4b59-bc5c-030ab5061fbf"). InnerVolumeSpecName "kube-api-access-mdpmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:26.454026 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.453988    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:26.454026 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.454022    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d86bcb95-5859-4b59-bc5c-030ab5061fbf-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:26.454026 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:26.454032    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdpmv\" (UniqueName: \"kubernetes.io/projected/d86bcb95-5859-4b59-bc5c-030ab5061fbf-kube-api-access-mdpmv\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:27.218492 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:27.218452    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78" event={"ID":"d86bcb95-5859-4b59-bc5c-030ab5061fbf","Type":"ContainerDied","Data":"eae318cde91df035e2a5ee2ee9cb1e433918f84a404b3bedfca8aefd3601156a"}
Oct 29 23:13:27.218637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:27.218531    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae318cde91df035e2a5ee2ee9cb1e433918f84a404b3bedfca8aefd3601156a"
Oct 29 23:13:27.218637 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:27.218468    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"
Oct 29 23:13:33.237206 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:13:33.237162    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:13:34.320112 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:34.320080    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"]
Oct 29 23:13:34.384091 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:34.384060    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:39.334193 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:39.334133    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:13:40.852338 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.852291    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"]
Oct 29 23:13:40.852839 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.852782    4839 cpu_manager.go:414] "containerMap: removing container" podUID="d86bcb95-5859-4b59-bc5c-030ab5061fbf" containerName="revoke-keys"
Oct 29 23:13:40.852839 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.852800    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86bcb95-5859-4b59-bc5c-030ab5061fbf" containerName="revoke-keys"
Oct 29 23:13:40.852967 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.852883    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86bcb95-5859-4b59-bc5c-030ab5061fbf" containerName="revoke-keys"
Oct 29 23:13:40.856089 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.856066    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:40.871274 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.871247    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:13:40.883631 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.883595    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:13:40.883790 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.883696    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:13:40.883790 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.883695    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:13:40.885718 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.885696    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"]
Oct 29 23:13:40.939459 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.939413    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twq2\" (UniqueName: \"kubernetes.io/projected/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-kube-api-access-8twq2\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:40.939459 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.939460    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:40.939727 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:40.939495    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.041123 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.041086    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twq2\" (UniqueName: \"kubernetes.io/projected/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-kube-api-access-8twq2\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.041123 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.041127    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.041353 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.041184    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.041816 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.041792    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.043510 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.043482    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.060828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.060798    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twq2\" (UniqueName: \"kubernetes.io/projected/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-kube-api-access-8twq2\") pod \"maas-api-revoke-sub-3e31a506-hwbsg\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.168973 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.168881    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:41.461783 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:13:41.461744    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105b7d47_600a_4fc8_94bb_d3c6898b7fbe.slice/crio-0267a4b137dde3302070b9df68304ecf4ecaa9ad9570edd0c4655acc618d3688 WatchSource:0}: Error finding container 0267a4b137dde3302070b9df68304ecf4ecaa9ad9570edd0c4655acc618d3688: Status 404 returned error can't find the container with id 0267a4b137dde3302070b9df68304ecf4ecaa9ad9570edd0c4655acc618d3688
Oct 29 23:13:41.462411 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:41.462387    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"]
Oct 29 23:13:42.303974 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:42.303938    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg" event={"ID":"105b7d47-600a-4fc8-94bb-d3c6898b7fbe","Type":"ContainerStarted","Data":"b960a7a3f86e1aab31a1a99f42cdc2d088ec49bd701047df2a30995dfa5fa394"}
Oct 29 23:13:42.304336 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:42.304005    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg" event={"ID":"105b7d47-600a-4fc8-94bb-d3c6898b7fbe","Type":"ContainerStarted","Data":"0267a4b137dde3302070b9df68304ecf4ecaa9ad9570edd0c4655acc618d3688"}
Oct 29 23:13:42.347230 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:42.347180    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg" podStartSLOduration=2.347165706 podStartE2EDuration="2.347165706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:13:42.346059729 +0000 UTC m=+1879.449136000" watchObservedRunningTime="2026-10-29 23:13:42.347165706 +0000 UTC m=+1879.450241968"
Oct 29 23:13:43.324930 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:43.324889    4839 generic.go:356] "Generic (PLEG): container finished" podID="105b7d47-600a-4fc8-94bb-d3c6898b7fbe" containerID="b960a7a3f86e1aab31a1a99f42cdc2d088ec49bd701047df2a30995dfa5fa394" exitCode=0
Oct 29 23:13:43.325480 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:43.324984    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg" event={"ID":"105b7d47-600a-4fc8-94bb-d3c6898b7fbe","Type":"ContainerDied","Data":"b960a7a3f86e1aab31a1a99f42cdc2d088ec49bd701047df2a30995dfa5fa394"}
Oct 29 23:13:45.076810 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.076516    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:45.162852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.153633    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-kube-api-access-8twq2\" (UniqueName: \"kubernetes.io/projected/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-kube-api-access-8twq2\") pod \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") "
Oct 29 23:13:45.162852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.157330    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-openshift-ca\") pod \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") "
Oct 29 23:13:45.162852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.157449    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-xks-ca\") pod \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\" (UID: \"105b7d47-600a-4fc8-94bb-d3c6898b7fbe\") "
Oct 29 23:13:45.162852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.160529    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-xks-ca" pod "105b7d47-600a-4fc8-94bb-d3c6898b7fbe" (UID: "105b7d47-600a-4fc8-94bb-d3c6898b7fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:13:45.162852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.160841    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-kube-api-access-8twq2" pod "105b7d47-600a-4fc8-94bb-d3c6898b7fbe" (UID: "105b7d47-600a-4fc8-94bb-d3c6898b7fbe"). InnerVolumeSpecName "kube-api-access-8twq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:45.162852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.161018    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-openshift-ca" pod "105b7d47-600a-4fc8-94bb-d3c6898b7fbe" (UID: "105b7d47-600a-4fc8-94bb-d3c6898b7fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:13:45.267546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.258763    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:45.267546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.258805    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:45.267546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.258820    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twq2\" (UniqueName: \"kubernetes.io/projected/105b7d47-600a-4fc8-94bb-d3c6898b7fbe-kube-api-access-8twq2\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:45.352828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.352793    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"
Oct 29 23:13:45.353104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.352843    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg" event={"ID":"105b7d47-600a-4fc8-94bb-d3c6898b7fbe","Type":"ContainerDied","Data":"0267a4b137dde3302070b9df68304ecf4ecaa9ad9570edd0c4655acc618d3688"}
Oct 29 23:13:45.353360 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:45.353339    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0267a4b137dde3302070b9df68304ecf4ecaa9ad9570edd0c4655acc618d3688"
Oct 29 23:13:49.339779 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.339732    4839 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79" podUID="2bd769a0-3d98-46c5-801a-6fe2328376c6" containerName="main" containerID="cri-o://647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae" gracePeriod=45
Oct 29 23:13:49.597471 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.597408    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:13:49.694657 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.694619    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd769a0-3d98-46c5-801a-6fe2328376c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd769a0-3d98-46c5-801a-6fe2328376c6-tls-certs\") pod \"2bd769a0-3d98-46c5-801a-6fe2328376c6\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") "
Oct 29 23:13:49.694828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.694674    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-home\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-home\") pod \"2bd769a0-3d98-46c5-801a-6fe2328376c6\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") "
Oct 29 23:13:49.694828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.694710    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-model-cache\") pod \"2bd769a0-3d98-46c5-801a-6fe2328376c6\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") "
Oct 29 23:13:49.694828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.694738    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-dshm\") pod \"2bd769a0-3d98-46c5-801a-6fe2328376c6\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") "
Oct 29 23:13:49.694828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.694763    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd769a0-3d98-46c5-801a-6fe2328376c6-kube-api-access-766r5\" (UniqueName: \"kubernetes.io/projected/2bd769a0-3d98-46c5-801a-6fe2328376c6-kube-api-access-766r5\") pod \"2bd769a0-3d98-46c5-801a-6fe2328376c6\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") "
Oct 29 23:13:49.694828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.694793    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-tmp-dir\") pod \"2bd769a0-3d98-46c5-801a-6fe2328376c6\" (UID: \"2bd769a0-3d98-46c5-801a-6fe2328376c6\") "
Oct 29 23:13:49.695120 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.694987    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-home" pod "2bd769a0-3d98-46c5-801a-6fe2328376c6" (UID: "2bd769a0-3d98-46c5-801a-6fe2328376c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:49.695120 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.695036    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-model-cache" pod "2bd769a0-3d98-46c5-801a-6fe2328376c6" (UID: "2bd769a0-3d98-46c5-801a-6fe2328376c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:49.695275 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.695175    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-tmp-dir" pod "2bd769a0-3d98-46c5-801a-6fe2328376c6" (UID: "2bd769a0-3d98-46c5-801a-6fe2328376c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:49.695275 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.695243    4839 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-home\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:49.695275 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.695258    4839 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-model-cache\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:49.696938 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.696911    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-dshm" pod "2bd769a0-3d98-46c5-801a-6fe2328376c6" (UID: "2bd769a0-3d98-46c5-801a-6fe2328376c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 29 23:13:49.697051 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.696973    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd769a0-3d98-46c5-801a-6fe2328376c6-tls-certs" pod "2bd769a0-3d98-46c5-801a-6fe2328376c6" (UID: "2bd769a0-3d98-46c5-801a-6fe2328376c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:13:49.697094 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.697052    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd769a0-3d98-46c5-801a-6fe2328376c6-kube-api-access-766r5" pod "2bd769a0-3d98-46c5-801a-6fe2328376c6" (UID: "2bd769a0-3d98-46c5-801a-6fe2328376c6"). InnerVolumeSpecName "kube-api-access-766r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:13:49.795818 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.795779    4839 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bd769a0-3d98-46c5-801a-6fe2328376c6-tls-certs\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:49.795818 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.795811    4839 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-dshm\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:49.795818 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.795820    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766r5\" (UniqueName: \"kubernetes.io/projected/2bd769a0-3d98-46c5-801a-6fe2328376c6-kube-api-access-766r5\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:49.796053 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:49.795829    4839 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bd769a0-3d98-46c5-801a-6fe2328376c6-tmp-dir\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:13:50.377551 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.377509    4839 generic.go:356] "Generic (PLEG): container finished" podID="2bd769a0-3d98-46c5-801a-6fe2328376c6" containerID="647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae" exitCode=0
Oct 29 23:13:50.378020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.377560    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79" event={"ID":"2bd769a0-3d98-46c5-801a-6fe2328376c6","Type":"ContainerDied","Data":"647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae"}
Oct 29 23:13:50.378020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.377580    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"
Oct 29 23:13:50.378020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.377601    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79" event={"ID":"2bd769a0-3d98-46c5-801a-6fe2328376c6","Type":"ContainerDied","Data":"7dcc02b53ebf78250a5a9cf587e2c1b525c501b1994457b77eda3a7b82e67ff7"}
Oct 29 23:13:50.378020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.377623    4839 scope.go:122] "RemoveContainer" containerID="647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae"
Oct 29 23:13:50.389105 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.389081    4839 scope.go:122] "RemoveContainer" containerID="647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae"
Oct 29 23:13:50.389703 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:13:50.389677    4839 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae\": container with ID starting with 647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae not found: ID does not exist" containerID="647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae"
Oct 29 23:13:50.389801 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.389714    4839 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae"} err="failed to get container status \"647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae\": rpc error: code = NotFound desc = could not find container \"647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae\": container with ID starting with 647053b8115b53a924bed8bec0a26c69aa40c0478f49096e8be2824d936b7eae not found: ID does not exist"
Oct 29 23:13:50.430261 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.430227    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"]
Oct 29 23:13:50.442847 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:50.442814    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-95f34a42-kserve-777c8c885-qfg79"]
Oct 29 23:13:51.007888 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:13:51.007849    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd769a0-3d98-46c5-801a-6fe2328376c6" path="/var/lib/kubelet/pods/2bd769a0-3d98-46c5-801a-6fe2328376c6/volumes"
Oct 29 23:14:03.237054 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:14:03.237017    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:14:06.287302 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:06.287260    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:11.279573 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.279536    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"]
Oct 29 23:14:11.279997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.279849    4839 cpu_manager.go:414] "containerMap: removing container" podUID="2bd769a0-3d98-46c5-801a-6fe2328376c6" containerName="main"
Oct 29 23:14:11.279997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.279860    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd769a0-3d98-46c5-801a-6fe2328376c6" containerName="main"
Oct 29 23:14:11.279997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.279881    4839 cpu_manager.go:414] "containerMap: removing container" podUID="105b7d47-600a-4fc8-94bb-d3c6898b7fbe" containerName="revoke-keys"
Oct 29 23:14:11.279997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.279886    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105b7d47-600a-4fc8-94bb-d3c6898b7fbe" containerName="revoke-keys"
Oct 29 23:14:11.279997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.279944    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd769a0-3d98-46c5-801a-6fe2328376c6" containerName="main"
Oct 29 23:14:11.279997 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.279954    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="105b7d47-600a-4fc8-94bb-d3c6898b7fbe" containerName="revoke-keys"
Oct 29 23:14:11.284421 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.284399    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.313033 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.313005    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 29 23:14:11.313033 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.313016    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:14:11.313268 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.313075    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:14:11.313268 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.313175    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tjz6s\""
Oct 29 23:14:11.315130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.315107    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"]
Oct 29 23:14:11.385453 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.385420    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crj54\" (UniqueName: \"kubernetes.io/projected/a859c9c4-76e6-46f0-94f8-5cd16a559e69-kube-api-access-crj54\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.385634 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.385534    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.385634 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.385591    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.486923 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.486885    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.487099 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.486933    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.487099 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.486973    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crj54\" (UniqueName: \"kubernetes.io/projected/a859c9c4-76e6-46f0-94f8-5cd16a559e69-kube-api-access-crj54\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.487648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.487623    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.489334 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.489309    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.506854 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.506822    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crj54\" (UniqueName: \"kubernetes.io/projected/a859c9c4-76e6-46f0-94f8-5cd16a559e69-kube-api-access-crj54\") pod \"maas-api-revoke-sub-b53f5837-sxqp9\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.597205 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.597110    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:11.888652 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:14:11.888610    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda859c9c4_76e6_46f0_94f8_5cd16a559e69.slice/crio-7a75e9fa71b0f7162fea0439200f20ec95560c153d3380aaeb136ff9340f4d80 WatchSource:0}: Error finding container 7a75e9fa71b0f7162fea0439200f20ec95560c153d3380aaeb136ff9340f4d80: Status 404 returned error can't find the container with id 7a75e9fa71b0f7162fea0439200f20ec95560c153d3380aaeb136ff9340f4d80
Oct 29 23:14:11.888838 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:11.888812    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"]
Oct 29 23:14:12.547439 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:12.547402    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9" event={"ID":"a859c9c4-76e6-46f0-94f8-5cd16a559e69","Type":"ContainerStarted","Data":"63a6874a2ddc4f944002a583d8fd8589929a80816755e1344c9eb86a2140e4d1"}
Oct 29 23:14:12.547872 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:12.547454    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9" event={"ID":"a859c9c4-76e6-46f0-94f8-5cd16a559e69","Type":"ContainerStarted","Data":"7a75e9fa71b0f7162fea0439200f20ec95560c153d3380aaeb136ff9340f4d80"}
Oct 29 23:14:12.589199 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:12.589129    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9" podStartSLOduration=1.589114338 podStartE2EDuration="1.589114338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:12.587281107 +0000 UTC m=+1909.690357396" watchObservedRunningTime="2026-10-29 23:14:12.589114338 +0000 UTC m=+1909.692190609"
Oct 29 23:14:13.555101 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:13.555065    4839 generic.go:356] "Generic (PLEG): container finished" podID="a859c9c4-76e6-46f0-94f8-5cd16a559e69" containerID="63a6874a2ddc4f944002a583d8fd8589929a80816755e1344c9eb86a2140e4d1" exitCode=0
Oct 29 23:14:13.555566 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:13.555159    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9" event={"ID":"a859c9c4-76e6-46f0-94f8-5cd16a559e69","Type":"ContainerDied","Data":"63a6874a2ddc4f944002a583d8fd8589929a80816755e1344c9eb86a2140e4d1"}
Oct 29 23:14:14.688551 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.688525    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:14.817403 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.817309    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a859c9c4-76e6-46f0-94f8-5cd16a559e69-kube-api-access-crj54\" (UniqueName: \"kubernetes.io/projected/a859c9c4-76e6-46f0-94f8-5cd16a559e69-kube-api-access-crj54\") pod \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") "
Oct 29 23:14:14.817403 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.817354    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-openshift-ca\") pod \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") "
Oct 29 23:14:14.817403 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.817384    4839 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-xks-ca\") pod \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\" (UID: \"a859c9c4-76e6-46f0-94f8-5cd16a559e69\") "
Oct 29 23:14:14.817709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.817686    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-openshift-ca" pod "a859c9c4-76e6-46f0-94f8-5cd16a559e69" (UID: "a859c9c4-76e6-46f0-94f8-5cd16a559e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 29 23:14:14.819576 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.819542    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a859c9c4-76e6-46f0-94f8-5cd16a559e69-kube-api-access-crj54" pod "a859c9c4-76e6-46f0-94f8-5cd16a559e69" (UID: "a859c9c4-76e6-46f0-94f8-5cd16a559e69"). InnerVolumeSpecName "kube-api-access-crj54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 29 23:14:14.819703 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.819599    4839 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-xks-ca" pod "a859c9c4-76e6-46f0-94f8-5cd16a559e69" (UID: "a859c9c4-76e6-46f0-94f8-5cd16a559e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 29 23:14:14.918616 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.918583    4839 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crj54\" (UniqueName: \"kubernetes.io/projected/a859c9c4-76e6-46f0-94f8-5cd16a559e69-kube-api-access-crj54\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:14:14.918616 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.918614    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-openshift-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:14:14.918826 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:14.918624    4839 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a859c9c4-76e6-46f0-94f8-5cd16a559e69-maas-api-xks-ca\") on node \"ip-10-0-136-187.ec2.internal\" DevicePath \"\""
Oct 29 23:14:15.566198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:15.566132    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9" event={"ID":"a859c9c4-76e6-46f0-94f8-5cd16a559e69","Type":"ContainerDied","Data":"7a75e9fa71b0f7162fea0439200f20ec95560c153d3380aaeb136ff9340f4d80"}
Oct 29 23:14:15.566198 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:15.566189    4839 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b53f5837-sxqp9"
Oct 29 23:14:15.566410 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:15.566217    4839 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a75e9fa71b0f7162fea0439200f20ec95560c153d3380aaeb136ff9340f4d80"
Oct 29 23:14:17.293167 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:17.293109    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:19.027104 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:19.027064    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"]
Oct 29 23:14:19.039528 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:19.039499    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gl8j7"]
Oct 29 23:14:21.006910 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:21.006876    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a09783-fa24-4d23-bbc5-c1ff6aadbada" path="/var/lib/kubelet/pods/c7a09783-fa24-4d23-bbc5-c1ff6aadbada/volumes"
Oct 29 23:14:23.592720 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.592658    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-gscmm"]
Oct 29 23:14:23.593208 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.593036    4839 cpu_manager.go:414] "containerMap: removing container" podUID="a859c9c4-76e6-46f0-94f8-5cd16a559e69" containerName="revoke-keys"
Oct 29 23:14:23.593208 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.593049    4839 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a859c9c4-76e6-46f0-94f8-5cd16a559e69" containerName="revoke-keys"
Oct 29 23:14:23.593208 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.593103    4839 memory_manager.go:365] "RemoveStaleState removing state" podUID="a859c9c4-76e6-46f0-94f8-5cd16a559e69" containerName="revoke-keys"
Oct 29 23:14:23.596279 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.596253    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.628297 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.628257    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-gscmm"]
Oct 29 23:14:23.691213 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.691181    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 29 23:14:23.691372 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.691296    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7l5lq\""
Oct 29 23:14:23.691372 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.691301    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 29 23:14:23.691372 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.691307    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 29 23:14:23.691372 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.691332    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 29 23:14:23.693678 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.693659    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a635d559-2b09-446c-903d-9cbbf57df2f2-maas-api-tls\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.693726 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.693697    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a635d559-2b09-446c-903d-9cbbf57df2f2-metrics-certs\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.693761 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.693747    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a635d559-2b09-446c-903d-9cbbf57df2f2-openshift-service-ca\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.693852 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.693834    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhfv\" (UniqueName: \"kubernetes.io/projected/a635d559-2b09-446c-903d-9cbbf57df2f2-kube-api-access-vvhfv\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.794948 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.794910    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a635d559-2b09-446c-903d-9cbbf57df2f2-maas-api-tls\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.795130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.794958    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a635d559-2b09-446c-903d-9cbbf57df2f2-metrics-certs\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.795130 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.795073    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a635d559-2b09-446c-903d-9cbbf57df2f2-openshift-service-ca\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.795255 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.795195    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhfv\" (UniqueName: \"kubernetes.io/projected/a635d559-2b09-446c-903d-9cbbf57df2f2-kube-api-access-vvhfv\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.795791 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.795767    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a635d559-2b09-446c-903d-9cbbf57df2f2-openshift-service-ca\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.797433 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.797412    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a635d559-2b09-446c-903d-9cbbf57df2f2-metrics-certs\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.797515 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.797456    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a635d559-2b09-446c-903d-9cbbf57df2f2-maas-api-tls\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.814914 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.814891    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhfv\" (UniqueName: \"kubernetes.io/projected/a635d559-2b09-446c-903d-9cbbf57df2f2-kube-api-access-vvhfv\") pod \"maas-api-595cc648c-gscmm\" (UID: \"a635d559-2b09-446c-903d-9cbbf57df2f2\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:23.909449 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:23.909357    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:24.205350 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:24.205317    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-gscmm"]
Oct 29 23:14:24.206091 ip-10-0-136-187 kubenswrapper[4839]: W1029 23:14:24.206064    4839 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda635d559_2b09_446c_903d_9cbbf57df2f2.slice/crio-69198d1350b042c8c6a99824e05c66307ffbab8153ed87cdd340861899b7cded WatchSource:0}: Error finding container 69198d1350b042c8c6a99824e05c66307ffbab8153ed87cdd340861899b7cded: Status 404 returned error can't find the container with id 69198d1350b042c8c6a99824e05c66307ffbab8153ed87cdd340861899b7cded
Oct 29 23:14:24.207500 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:24.207480    4839 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 29 23:14:24.610347 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:24.610307    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm" event={"ID":"a635d559-2b09-446c-903d-9cbbf57df2f2","Type":"ContainerStarted","Data":"69198d1350b042c8c6a99824e05c66307ffbab8153ed87cdd340861899b7cded"}
Oct 29 23:14:24.670256 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:24.670226    4839 scope.go:122] "RemoveContainer" containerID="43b5a382d4615aa0bae81965c118262f3c73e40ceeb4c873750d647942153ecd"
Oct 29 23:14:25.617234 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:25.617192    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm" event={"ID":"a635d559-2b09-446c-903d-9cbbf57df2f2","Type":"ContainerStarted","Data":"b73fd4284a654cd1739feb96bebb02d562d83bf080469fe9bd90b6f7f6d5bf5a"}
Oct 29 23:14:25.617660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:25.617309    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:25.656760 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:25.656685    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm" podStartSLOduration=2.126294847 podStartE2EDuration="2.65667121s" totalImagesPullingTime="530.376363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-29 23:14:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:14:25.655750731 +0000 UTC m=+1922.758827002" watchObservedRunningTime="2026-10-29 23:14:25.65667121 +0000 UTC m=+1922.759747480"
Oct 29 23:14:31.628467 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:31.628436    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-gscmm"
Oct 29 23:14:33.251470 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:14:33.251399    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:14:34.905308 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:34.905262    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-snxnt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:14:34.905902 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:34.905865    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-snxnt" podUID="5b0c4083-358e-4fd3-864b-db6b53a4990e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:35.222391 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:35.222278    4839 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 29 23:14:35.222703 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:35.222650    4839 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hqqmd" podUID="e42f233c-1cec-4162-abdb-2355ee0fb03a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 29 23:14:45.131611 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:45.131542    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:14:45.232700 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:14:45.232663    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:03.237563 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:15:03.237519    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:15:17.418559 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:15:17.418519    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:24.693730 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:15:24.693693    4839 scope.go:122] "RemoveContainer" containerID="3d6dfd6aa3522935e5690fc9b74677bdc823eb94d179b9715eb7c3f794a96f3e"
Oct 29 23:15:32.090349 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:15:32.090317    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:32.179069 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:15:32.179034    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:15:33.237257 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:15:33.237210    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:16:03.237201 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:16:03.237137    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:16:09.031556 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:09.031478    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"]
Oct 29 23:16:09.049386 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:09.044097    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-cwkfv"]
Oct 29 23:16:09.230660 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:09.230621    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d74798bcb-jpzf4_0be7d699-ee3d-4d60-a663-c7a8730407ec/manager/0.log"
Oct 29 23:16:10.806379 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:10.806350    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-n78vt_91cd164a-636c-4878-b004-af3bd0eda83d/manager/0.log"
Oct 29 23:16:11.007399 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:11.007365    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8289e483-ab28-425a-be97-a0b67295bbf6" path="/var/lib/kubelet/pods/8289e483-ab28-425a-be97-a0b67295bbf6/volumes"
Oct 29 23:16:11.050401 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:11.050373    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-ctkl2_19f65fc3-d04a-4a71-b3d2-9a27f001912c/copy-manifests/0.log"
Oct 29 23:16:11.085619 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:11.085552    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-ctkl2_19f65fc3-d04a-4a71-b3d2-9a27f001912c/manager/0.log"
Oct 29 23:16:12.569063 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:12.569031    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-gscmm_a635d559-2b09-446c-903d-9cbbf57df2f2/maas-api/0.log"
Oct 29 23:16:13.130526 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:13.130499    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-hwbsg_105b7d47-600a-4fc8-94bb-d3c6898b7fbe/revoke-keys/0.log"
Oct 29 23:16:13.319830 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:13.319800    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-h4g78_d86bcb95-5859-4b59-bc5c-030ab5061fbf/revoke-keys/0.log"
Oct 29 23:16:13.879633 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:13.879604    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b53f5837-sxqp9_a859c9c4-76e6-46f0-94f8-5cd16a559e69/revoke-keys/0.log"
Oct 29 23:16:14.070530 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:14.070499    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-hxcdj_a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e/revoke-keys/0.log"
Oct 29 23:16:15.784746 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:15.784694    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:15.867590 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:15.867539    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:17.538051 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:17.538002    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:18.298338 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:18.298283    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lbn4t_0660ba50-ece7-4b6d-a7e1-223cd5a72603/discovery/0.log"
Oct 29 23:16:18.702410 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:18.702314    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d6d655bb-nssk5_8f8469ab-1187-4d92-9399-863ea18599f9/kube-auth-proxy/0.log"
Oct 29 23:16:18.882903 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:18.882875    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-84ct6_f4957a03-1161-47a8-a4c1-69c75f901597/istio-proxy/0.log"
Oct 29 23:16:19.867709 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:19.867673    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-hqqmd_e42f233c-1cec-4162-abdb-2355ee0fb03a/istio-proxy/0.log"
Oct 29 23:16:20.078819 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:20.078772    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-rntkv_80f4fedc-6afb-42e4-87de-8b77868b9125/istio-proxy/0.log"
Oct 29 23:16:20.694561 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:20.694518    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-snxnt_5b0c4083-358e-4fd3-864b-db6b53a4990e/istio-proxy/0.log"
Oct 29 23:16:21.088189 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:21.088126    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-nthkr_00bbe52f-800c-4843-a6ab-9ce20cb3a508/payload-processing/0.log"
Oct 29 23:16:22.655014 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:22.654979    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-t7hn5_21d37309-aa51-44b7-8fdd-e87e114aaaad/main/0.log"
Oct 29 23:16:23.042302 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:23.042261    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h_6b1ce2aa-79c5-4963-84b2-993a8a24dfcb/main/0.log"
Oct 29 23:16:24.028335 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:24.028294    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"]
Oct 29 23:16:24.039993 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:24.039961    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hxcdj"]
Oct 29 23:16:25.010259 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:25.010222    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e" path="/var/lib/kubelet/pods/a0c8d3cc-66ea-4b5e-ae49-f34da9cc1f6e/volumes"
Oct 29 23:16:33.236853 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:16:33.236812    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:16:33.513536 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:33.513461    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-d74798bcb-jpzf4_0be7d699-ee3d-4d60-a663-c7a8730407ec/manager/0.log"
Oct 29 23:16:35.054051 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:35.054018    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-n78vt_91cd164a-636c-4878-b004-af3bd0eda83d/manager/0.log"
Oct 29 23:16:35.314823 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:35.314748    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-ctkl2_19f65fc3-d04a-4a71-b3d2-9a27f001912c/copy-manifests/0.log"
Oct 29 23:16:35.351784 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:35.351754    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-675b698459-ctkl2_19f65fc3-d04a-4a71-b3d2-9a27f001912c/manager/0.log"
Oct 29 23:16:36.603020 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:36.602990    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-gscmm_a635d559-2b09-446c-903d-9cbbf57df2f2/maas-api/0.log"
Oct 29 23:16:37.197633 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:37.197605    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-hwbsg_105b7d47-600a-4fc8-94bb-d3c6898b7fbe/revoke-keys/0.log"
Oct 29 23:16:37.399507 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:37.399472    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-h4g78_d86bcb95-5859-4b59-bc5c-030ab5061fbf/revoke-keys/0.log"
Oct 29 23:16:37.963391 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:37.963357    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b53f5837-sxqp9_a859c9c4-76e6-46f0-94f8-5cd16a559e69/revoke-keys/0.log"
Oct 29 23:16:41.528743 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:41.528708    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lbn4t_0660ba50-ece7-4b6d-a7e1-223cd5a72603/discovery/0.log"
Oct 29 23:16:41.957206 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:41.957113    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d6d655bb-nssk5_8f8469ab-1187-4d92-9399-863ea18599f9/kube-auth-proxy/0.log"
Oct 29 23:16:42.149271 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:42.149225    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-84ct6_f4957a03-1161-47a8-a4c1-69c75f901597/istio-proxy/0.log"
Oct 29 23:16:43.210332 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:43.210301    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-hqqmd_e42f233c-1cec-4162-abdb-2355ee0fb03a/istio-proxy/0.log"
Oct 29 23:16:43.417924 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:43.417897    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-rntkv_80f4fedc-6afb-42e4-87de-8b77868b9125/istio-proxy/0.log"
Oct 29 23:16:44.104042 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:44.104007    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-snxnt_5b0c4083-358e-4fd3-864b-db6b53a4990e/istio-proxy/0.log"
Oct 29 23:16:44.487546 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:44.487456    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-nthkr_00bbe52f-800c-4843-a6ab-9ce20cb3a508/payload-processing/0.log"
Oct 29 23:16:45.601244 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:45.601203    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-t7hn5_21d37309-aa51-44b7-8fdd-e87e114aaaad/main/0.log"
Oct 29 23:16:46.003369 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:46.003284    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b82sc2h_6b1ce2aa-79c5-4963-84b2-993a8a24dfcb/main/0.log"
Oct 29 23:16:49.957876 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:49.957838    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:16:50.094708 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:16:50.094674    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:03.237111 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:17:03.237059    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:17:09.390506 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:17:09.390448    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:17.659407 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:17:17.659356    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:17:24.719945 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:17:24.719911    4839 scope.go:122] "RemoveContainer" containerID="8e796ba177bd2fecb68ba1e210b2dddf85851bc8340bf83b7d28af5d57a3cbfe"
Oct 29 23:17:24.728604 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:17:24.728579    4839 scope.go:122] "RemoveContainer" containerID="9e5c8f71de6ad986f42c52a0be3f68e55e1f3a3e4e332ca2629a3e7bb6ee0ac9"
Oct 29 23:17:25.768057 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:17:25.768019    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-nthkr_00bbe52f-800c-4843-a6ab-9ce20cb3a508/payload-processing/0.log"
Oct 29 23:17:25.951512 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:17:25.951482    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-nthkr_00bbe52f-800c-4843-a6ab-9ce20cb3a508/payload-processing/0.log"
Oct 29 23:17:33.238065 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:17:33.238027    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:18:03.237472 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:18:03.237390    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:18:13.987206 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:13.987172    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-nthkr_00bbe52f-800c-4843-a6ab-9ce20cb3a508/payload-processing/0.log"
Oct 29 23:18:14.177254 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:14.177215    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-nthkr_00bbe52f-800c-4843-a6ab-9ce20cb3a508/payload-processing/0.log"
Oct 29 23:18:14.533705 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:14.533665    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:14.673609 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:14.673573    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:17.784335 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:17.784296    4839 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log" limit=1024
Oct 29 23:18:19.294486 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:19.294459    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cn2r6_0fe38f83-f940-4e6a-8f45-0148745b3693/global-pull-secret-syncer/0.log"
Oct 29 23:18:19.851438 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:19.851403    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dwfrq_9c6637f1-cc56-4ad1-b1dd-d7ed5e3bb2e8/konnectivity-agent/0.log"
Oct 29 23:18:20.274219 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:20.274189    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-187.ec2.internal_5a74b888ca6cfc8b230ed1966a77b369/haproxy/0.log"
Oct 29 23:18:27.031589 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:27.031553    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"]
Oct 29 23:18:27.043881 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:27.043849    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h4g78"]
Oct 29 23:18:28.592268 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.592231    4839 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"]
Oct 29 23:18:28.595828 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.595804    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.612753 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.612727    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"]
Oct 29 23:18:28.684326 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.684296    4839 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rbmpv\"/\"default-dockercfg-wghv7\""
Oct 29 23:18:28.701902 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.701870    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmpv\"/\"openshift-service-ca.crt\""
Oct 29 23:18:28.702043 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.701916    4839 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rbmpv\"/\"kube-root-ca.crt\""
Oct 29 23:18:28.731465 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.731423    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-proc\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.731607 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.731486    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9fq6\" (UniqueName: \"kubernetes.io/projected/d8de04cd-4096-484e-8206-9c5d077ede06-kube-api-access-r9fq6\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.731607 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.731592    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-podres\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.731738 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.731652    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-sys\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.731779 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.731744    4839 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-lib-modules\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833193 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833125    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9fq6\" (UniqueName: \"kubernetes.io/projected/d8de04cd-4096-484e-8206-9c5d077ede06-kube-api-access-r9fq6\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833402 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833206    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-podres\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833402 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833268    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-sys\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833402 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833341    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-lib-modules\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833402 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833343    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-podres\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833402 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833373    4839 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-proc\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833402 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833343    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-sys\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833638 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833421    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-proc\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.833638 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.833464    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8de04cd-4096-484e-8206-9c5d077ede06-lib-modules\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.853521 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.853453    4839 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9fq6\" (UniqueName: \"kubernetes.io/projected/d8de04cd-4096-484e-8206-9c5d077ede06-kube-api-access-r9fq6\") pod \"perf-node-gather-daemonset-z6qkq\" (UID: \"d8de04cd-4096-484e-8206-9c5d077ede06\") " pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:28.908162 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:28.908114    4839 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:29.007057 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:29.007018    4839 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86bcb95-5859-4b59-bc5c-030ab5061fbf" path="/var/lib/kubelet/pods/d86bcb95-5859-4b59-bc5c-030ab5061fbf/volumes"
Oct 29 23:18:29.202159 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:29.202120    4839 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"]
Oct 29 23:18:29.844102 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:29.844067    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq" event={"ID":"d8de04cd-4096-484e-8206-9c5d077ede06","Type":"ContainerStarted","Data":"40925648cdf3f85cebc9eb51b7d3ede97a8d94cd60e2707092c7d61f5a6b75f6"}
Oct 29 23:18:29.844547 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:29.844131    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:29.844547 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:29.844175    4839 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq" event={"ID":"d8de04cd-4096-484e-8206-9c5d077ede06","Type":"ContainerStarted","Data":"e3d2677616e46326e7c41b38d81b2f2603674d70b692f77dc79fdf977446df4f"}
Oct 29 23:18:29.885648 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:29.885596    4839 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq" podStartSLOduration=1.885582068 podStartE2EDuration="1.885582068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-29 23:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-29 23:18:29.884432546 +0000 UTC m=+2166.987508817" watchObservedRunningTime="2026-10-29 23:18:29.885582068 +0000 UTC m=+2166.988658339"
Oct 29 23:18:30.352793 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:30.352761    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jfhv_6ac72c66-33a8-476a-afc7-093dc998aa4e/node-exporter/0.log"
Oct 29 23:18:30.469067 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:30.469043    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jfhv_6ac72c66-33a8-476a-afc7-093dc998aa4e/kube-rbac-proxy/0.log"
Oct 29 23:18:30.587292 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:30.587265    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6jfhv_6ac72c66-33a8-476a-afc7-093dc998aa4e/init-textfile/0.log"
Oct 29 23:18:33.237655 ip-10-0-136-187 kubenswrapper[4839]: E1029 23:18:33.237618    4839 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kf7bp" containerName="extractor"
Oct 29 23:18:35.858825 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:35.858791    4839 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rbmpv/perf-node-gather-daemonset-z6qkq"
Oct 29 23:18:41.451618 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:41.451587    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-692j2_49d13647-137d-4aff-a28b-705df3a5b728/dns/0.log"
Oct 29 23:18:41.569609 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:41.569580    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-692j2_49d13647-137d-4aff-a28b-705df3a5b728/kube-rbac-proxy/0.log"
Oct 29 23:18:42.293808 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:42.293776    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rhshl_782ad8cb-0365-4cb1-86b0-4d3059401b91/dns-node-resolver/0.log"
Oct 29 23:18:42.885623 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:42.885586    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-lbn4t_0660ba50-ece7-4b6d-a7e1-223cd5a72603/discovery/0.log"
Oct 29 23:18:42.981647 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:42.981619    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-84d6d655bb-nssk5_8f8469ab-1187-4d92-9399-863ea18599f9/kube-auth-proxy/0.log"
Oct 29 23:18:43.046570 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:43.046538    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-84ct6_f4957a03-1161-47a8-a4c1-69c75f901597/istio-proxy/0.log"
Oct 29 23:18:43.364680 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:43.364646    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-hqqmd_e42f233c-1cec-4162-abdb-2355ee0fb03a/istio-proxy/0.log"
Oct 29 23:18:43.427673 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:43.427641    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-rntkv_80f4fedc-6afb-42e4-87de-8b77868b9125/istio-proxy/0.log"
Oct 29 23:18:43.626457 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:43.626373    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-snxnt_5b0c4083-358e-4fd3-864b-db6b53a4990e/istio-proxy/0.log"
Oct 29 23:18:43.748009 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:43.747982    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-6bfdff5577-nthkr_00bbe52f-800c-4843-a6ab-9ce20cb3a508/payload-processing/0.log"
Oct 29 23:18:43.839342 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:43.839312    4839 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qxjxd_e3ec7504-ce41-4598-b6e6-1c8706119621/node-ca/0.log"
Oct 29 23:18:45.028193 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:45.028142    4839 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"]
Oct 29 23:18:45.041032 ip-10-0-136-187 kubenswrapper[4839]: I1029 23:18:45.041002    4839 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hwbsg"]