Jul 29 20:47:29.153155 ip-10-0-136-64 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 20:47:29.574692 ip-10-0-136-64 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:47:29.574692 ip-10-0-136-64 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 20:47:29.574692 ip-10-0-136-64 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:47:29.574692 ip-10-0-136-64 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 20:47:29.574692 ip-10-0-136-64 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 20:47:29.576709 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.576621    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 20:47:29.584140 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584118    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:47:29.584140 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584136    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:47:29.584140 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584140    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:47:29.584140 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584143    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584157    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584160    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584165    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584170    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584174    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584178    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584181    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584184    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584188    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584190    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584199    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584202    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584208    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584210    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584213    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584216    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584218    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584221    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:47:29.584321 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584223    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584226    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584228    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584231    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584234    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584237    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584239    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584244    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584247    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584250    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584252    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584255    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584257    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584260    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584263    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584269    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584273    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584276    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584282    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584284    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:47:29.584780 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584287    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584290    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584293    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584295    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584325    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584336    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584339    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584343    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584346    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584349    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584352    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584357    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584360    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584363    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584366    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584368    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584371    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584374    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584377    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584380    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:47:29.585318 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584382    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584385    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584388    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584391    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584396    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584398    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584401    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584404    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584407    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584409    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584412    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584415    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584418    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584420    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584423    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584426    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584431    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584433    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584441    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584445    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:47:29.585811 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584448    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584450    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584453    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584456    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584460    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584463    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584466    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584469    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584475    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584478    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584481    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584484    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584487    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584490    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584493    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584496    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.584498    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585719    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585727    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:47:29.586325 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585730    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585736    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585739    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585742    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585744    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585747    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585750    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585753    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585755    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585758    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585761    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585764    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585773    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585775    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585781    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585784    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585786    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585789    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585792    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585794    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:47:29.586785 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585797    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585799    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585803    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585805    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585808    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585811    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585816    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585819    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585822    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585824    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585827    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585831    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585837    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585840    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585843    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585846    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585849    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585851    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585854    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585860    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:47:29.587294 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585863    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585866    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585868    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585871    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585873    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585896    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585899    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585902    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585905    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585908    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585920    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585925    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585928    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585930    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585933    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585937    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585940    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585943    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585946    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:47:29.587810 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585949    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585951    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585954    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585956    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585962    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585967    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585970    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585973    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585976    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585979    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585982    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585985    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585987    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585990    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585992    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585995    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.585998    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586003    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586005    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586014    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:47:29.588286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586019    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586021    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586024    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586028    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586030    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586033    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586036    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586039    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586045    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586047    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586050    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586053    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586055    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586058    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586061    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586063    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586066    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586069    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586771    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586776    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:47:29.588782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586779    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586782    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586784    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586787    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586792    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586795    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586798    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586801    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586804    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586806    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586809    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586817    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586820    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586822    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586825    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586828    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586831    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586836    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586839    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586842    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:47:29.589286 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586845    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586847    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586850    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586853    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586855    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586860    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586863    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586866    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586868    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586874    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586876    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586892    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586895    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586898    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586900    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586903    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586905    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586908    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586911    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586913    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:47:29.589782 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586916    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586918    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586923    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586926    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586934    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586937    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586940    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586943    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586946    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586948    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586951    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586954    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586957    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586959    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586970    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586973    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586976    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586979    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586982    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586984    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:47:29.590303 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586987    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586989    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586992    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586994    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.586997    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587000    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587004    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587007    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587010    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587012    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587015    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587017    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587020    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587022    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587025    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587027    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587030    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587038    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587041    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587046    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:47:29.590792 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587049    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587051    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587054    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587056    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587061    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587064    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587069    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587072    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587076    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587081    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587087    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587090    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587093    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587097    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587100    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587103    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.587106    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587330    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587350    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 20:47:29.591301 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587474    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587479    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587484    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587487    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587492    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587496    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587500    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587503    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587506    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587510    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587513    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587516    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587519    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587523    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587526    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587529    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587534    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587537    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587540    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587543    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587546    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587557    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587560    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 20:47:29.591773 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587564    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587567    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587570    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587573    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587576    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587579    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587582    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587586    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587592    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587597    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587600    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587605    2579 flags.go:64] FLAG: --enable-server="true"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587608    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587615    2579 flags.go:64] FLAG: --event-burst="100"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587620    2579 flags.go:64] FLAG: --event-qps="50"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587623    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587626    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587631    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587635    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587640    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587645    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587650    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587653    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587657    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587662    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 20:47:29.592346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587665    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587669    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587672    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587676    2579 flags.go:64] FLAG: --feature-gates=""
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587682    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587685    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587689    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587693    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587696    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587700    2579 flags.go:64] FLAG: --help="false"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587705    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-64.ec2.internal"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587708    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587711    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587714    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587718    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587723    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587726    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587729    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587732    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587735    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587739    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587742    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587745    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587748    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 20:47:29.592987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587751    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587754    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587757    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587760    2579 flags.go:64] FLAG: --lock-file=""
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587763    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587766    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587768    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587774    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587784    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587787    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587789    2579 flags.go:64] FLAG: --logging-format="text"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587792    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587796    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587798    2579 flags.go:64] FLAG: --manifest-url=""
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587802    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587806    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587810    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587814    2579 flags.go:64] FLAG: --max-pods="110"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587817    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587820    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587823    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587826    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587829    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587833    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587836    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 20:47:29.593624 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587845    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587848    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587851    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587854    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587857    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587862    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587865    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587868    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587871    2579 flags.go:64] FLAG: --port="10250"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587874    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587877    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a861fab3f43d3ea8"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587892    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587895    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587898    2579 flags.go:64] FLAG: --register-node="true"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587901    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587909    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587918    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587921    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587924    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587927    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587931    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587933    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587937    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587939    2579 flags.go:64] FLAG: --runonce="false"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587942    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 20:47:29.594279 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587946    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587948    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587951    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587954    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587957    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587961    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587965    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587968    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587971    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587974    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587977    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587980    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587983    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587985    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587992    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587995    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.587998    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588002    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588004    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588007    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588010    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588013    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588016    2579 flags.go:64] FLAG: --v="2"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588020    2579 flags.go:64] FLAG: --version="false"
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588030    2579 flags.go:64] FLAG: --vmodule=""
Jul 29 20:47:29.594912 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588034    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.588037    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588167    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588171    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588174    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588177    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588180    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588183    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588186    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588189    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588191    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588195    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588199    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588203    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588205    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588208    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588211    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588213    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588215    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588218    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588220    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:47:29.595551 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588223    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588226    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588228    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588231    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588234    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588236    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588239    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588241    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588244    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588246    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588249    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588259    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588261    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588264    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588266    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588269    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588271    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588274    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588278    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588281    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:47:29.596076 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588284    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588287    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588290    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588294    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588298    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588300    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588303    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588305    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588308    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588311    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588313    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588315    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588318    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588321    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588323    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588326    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588328    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588331    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588334    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:47:29.596576 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588336    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588339    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588342    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588344    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588347    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588355    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588357    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588360    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588363    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588365    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588367    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588370    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588372    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588375    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588378    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588380    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588388    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588391    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588394    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:47:29.597084 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588398    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588402    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588404    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588407    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588410    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588413    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588416    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588418    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588421    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588424    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588426    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588428    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588431    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588433    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588436    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588439    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588443    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588445    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588448    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588457    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:47:29.597558 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588460    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:47:29.598068 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.588462    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:47:29.598068 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.589459    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:47:29.598068 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.597874    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 20:47:29.598068 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.597904    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 20:47:29.598759 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.598747    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 20:47:29.598790 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.598759    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598814    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598819    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598823    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598826    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598829    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598833    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598835    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598838    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598841    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598844    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598847    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598849    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598852    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598855    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:47:29.598851 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598858    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598861    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598864    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598866    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598869    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598872    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598875    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598877    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598894    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598897    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598900    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598902    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598905    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598908    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598910    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598914    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598917    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598919    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598922    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598924    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:47:29.599329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598927    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598929    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598932    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598934    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598937    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598939    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598944    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598948    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598951    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598953    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598956    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598959    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598963    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598966    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598968    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598971    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598974    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598977    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598980    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598982    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:47:29.599827 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598985    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598987    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598990    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598992    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598995    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.598998    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599001    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599005    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599009    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599012    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599015    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599019    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599023    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599027    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599029    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599032    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599035    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599037    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599040    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:47:29.600329 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599043    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599045    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599048    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599050    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599053    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599056    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599059    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599062    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599064    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599067    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599070    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599073    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599075    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599078    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599080    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599083    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599086    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599089    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599091    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:47:29.600798 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599094    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599096    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599099    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599102    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599104    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599107    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599109    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.599114    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599234    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599239    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599242    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599245    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599248    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599251    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599253    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 20:47:29.601295 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599256    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599258    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599261    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599264    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599267    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599270    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599273    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599276    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599279    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599282    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599284    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599287    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599290    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599292    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599295    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599297    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599300    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599303    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599305    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 20:47:29.601660 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599308    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599311    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599313    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599316    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599318    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599321    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599325    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599328    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599331    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599334    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599336    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599339    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599342    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599345    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599347    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599350    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599353    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599356    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599359    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599362    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 20:47:29.602147 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599365    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599367    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599370    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599372    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599375    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599377    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599380    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599383    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599385    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599388    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599390    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599392    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599395    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599398    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599400    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599404    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599406    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599409    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599411    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599414    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 20:47:29.602627 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599416    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599419    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599421    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599424    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599426    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599429    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599432    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599435    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599438    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599441    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599444    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599446    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599449    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599451    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599454    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599456    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599459    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599461    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599464    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599466    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 20:47:29.603127 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599469    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599471    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599474    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599476    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599479    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599482    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599484    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599486    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599489    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599492    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599494    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599498    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:29.599500    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.599505    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 20:47:29.603602 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.599608    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 20:47:29.603976 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.603293    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 20:47:29.604778 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.604765    2579 server.go:1019] "Starting client certificate rotation"
Jul 29 20:47:29.604876 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.604861    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 20:47:29.604995 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.604908    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 20:47:29.608312 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.608300    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 20:47:29.611938 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.611912    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 20:47:29.626847 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.626826    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 29 20:47:29.628921 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.628902    2579 log.go:25] "Validated CRI v1 image API"
Jul 29 20:47:29.630388 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.630376    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 20:47:29.632776 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.632747    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a770f574-ac01-4851-9c04-571b737af5f8:/dev/nvme0n1p4 f6706e21-d878-4690-b355-6e44f9d2727b:/dev/nvme0n1p3]
Jul 29 20:47:29.632817 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.632777    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 20:47:29.640276 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.640163    2579 manager.go:216] Machine: {Timestamp:2026-07-29 20:47:29.638827404 +0000 UTC m=+0.360722499 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100002 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b5b2d3c8617b2499a48b2645a5612 SystemUUID:ec2b5b2d-3c86-17b2-499a-48b2645a5612 BootID:c89bc30c-7ed3-448c-9b3a-1eb1b1f1dab4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:67:81:d5:15 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:67:81:d5:15 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:77:36:ab:f6:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 20:47:29.640276 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.640260    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 20:47:29.640423 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.640372    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 20:47:29.643722 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.643694    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 20:47:29.643857 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.643724    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 20:47:29.644507 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.644497    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 20:47:29.644539 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.644509    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 20:47:29.644539 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.644517    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 20:47:29.645069 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.645053    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 20:47:29.645179 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.645167    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 20:47:29.646155 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.646145    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 20:47:29.648336 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648324    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 20:47:29.648461 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648452    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 20:47:29.648500 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648483    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 20:47:29.648500 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648495    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 20:47:29.648561 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648509    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 20:47:29.648561 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648519    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 29 20:47:29.648561 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648527    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 20:47:29.648917 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648905    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 20:47:29.648964 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.648926    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 20:47:29.651259 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.651239    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 20:47:29.653328 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.653311    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 20:47:29.653417 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.653350    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 20:47:29.656046 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656031    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656051    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656060    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656068    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656082    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656091    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656102    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656111    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 20:47:29.656121 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656122    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 20:47:29.656370 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656131    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 20:47:29.656370 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656155    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 20:47:29.656370 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.656164    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 20:47:29.657015 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.657003    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 20:47:29.657075 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.657017    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 20:47:29.658768 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.658741    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 20:47:29.660394 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.660371    2579 server.go:1268] "Started kubelet"
Jul 29 20:47:29.660488 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.660468    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 20:47:29.660629 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.660546    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 20:47:29.660797 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.660769    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 20:47:29.660929 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.660526    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 20:47:29.661249 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.660653    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 20:47:29.661306 ip-10-0-136-64 systemd[1]: Started Kubernetes Kubelet.
Jul 29 20:47:29.663135 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.663117    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 20:47:29.663949 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.663934    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 20:47:29.665576 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.665556    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 20:47:29.668654 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.667470    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18c6ddafdae3a483  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-07-29 20:47:29.660347523 +0000 UTC m=+0.382242617,LastTimestamp:2026-07-29 20:47:29.660347523 +0000 UTC m=+0.382242617,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Jul 29 20:47:29.670655 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.670641    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 20:47:29.670655 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.670647    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 20:47:29.670765 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.670703    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:29.672131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.671713    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 20:47:29.672131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.671751    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 20:47:29.672131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.671966    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 20:47:29.672131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.672080    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 20:47:29.672131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.672093    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 20:47:29.672710 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.672522    2579 factory.go:55] Registering systemd factory
Jul 29 20:47:29.672710 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.672544    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 29 20:47:29.673244 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.673224    2579 factory.go:153] Registering CRI-O factory
Jul 29 20:47:29.673244 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.673246    2579 factory.go:223] Registration of the crio container factory successfully
Jul 29 20:47:29.673382 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.673322    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 20:47:29.673382 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.673347    2579 factory.go:103] Registering Raw factory
Jul 29 20:47:29.673382 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.673363    2579 manager.go:1196] Started watching for new ooms in manager
Jul 29 20:47:29.674114 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.674098    2579 manager.go:349] Starting recovery of all containers
Jul 29 20:47:29.678027 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.678001    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 20:47:29.678246 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.678223    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 20:47:29.688005 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.687980    2579 manager.go:354] Recovery completed
Jul 29 20:47:29.692166 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.692151    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:29.694753 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.694734    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:29.694830 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.694787    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:29.694830 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.694804    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:29.695346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.695333    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 20:47:29.695398 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.695346    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 20:47:29.695398 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.695366    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 20:47:29.695398 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.695389    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:29.696210 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.696195    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:29.696295 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.696257    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:29.696295 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.696273    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:29.697116 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.697032    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18c6ddafdcf0a295  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-64.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-07-29 20:47:29.694753429 +0000 UTC m=+0.416648527,LastTimestamp:2026-07-29 20:47:29.694753429 +0000 UTC m=+0.416648527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Jul 29 20:47:29.697658 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.697646    2579 policy_none.go:49] "None policy: Start"
Jul 29 20:47:29.697712 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.697662    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 20:47:29.697712 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.697674    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 20:47:29.698978 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.698967    2579 policy_none.go:47] "Start"
Jul 29 20:47:29.706526 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.706448    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18c6ddafdcf146e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-64.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-07-29 20:47:29.694795496 +0000 UTC m=+0.416690598,LastTimestamp:2026-07-29 20:47:29.694795496 +0000 UTC m=+0.416690598,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Jul 29 20:47:29.716173 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.716111    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18c6ddafdcf180d9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-64.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-07-29 20:47:29.694810329 +0000 UTC m=+0.416705426,LastTimestamp:2026-07-29 20:47:29.694810329 +0000 UTC m=+0.416705426,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Jul 29 20:47:29.723197 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.723174    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58wsm"
Jul 29 20:47:29.725343 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.725169    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-64.ec2.internal.18c6ddafdcf0a295\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18c6ddafdcf0a295  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-64.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-07-29 20:47:29.694753429 +0000 UTC m=+0.416648527,LastTimestamp:2026-07-29 20:47:29.696214475 +0000 UTC m=+0.418109572,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Jul 29 20:47:29.733517 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.733502    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-58wsm"
Jul 29 20:47:29.735658 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.735597    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-64.ec2.internal.18c6ddafdcf146e8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-64.ec2.internal.18c6ddafdcf146e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-64.ec2.internal,UID:ip-10-0-136-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-64.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-64.ec2.internal,},FirstTimestamp:2026-07-29 20:47:29.694795496 +0000 UTC m=+0.416690598,LastTimestamp:2026-07-29 20:47:29.696265225 +0000 UTC m=+0.418160319,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-64.ec2.internal,}"
Jul 29 20:47:29.750509 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.750486    2579 manager.go:337] "Starting Device Plugin manager"
Jul 29 20:47:29.750610 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.750524    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 20:47:29.750610 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.750537    2579 server.go:87] "Starting device plugin registration server"
Jul 29 20:47:29.750798 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.750785    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 20:47:29.750852 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.750799    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 20:47:29.750923 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.750896    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 20:47:29.751001 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.750975    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 20:47:29.751001 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.750989    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 20:47:29.751531 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.751508    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 20:47:29.751607 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.751558    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:29.842673 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.842580    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 20:47:29.844159 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.843834    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 20:47:29.844159 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.843899    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 20:47:29.844159 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.843927    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 20:47:29.844159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.843968    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 20:47:29.846415 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.846397    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:47:29.850947 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.850928    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:29.851740 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.851724    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:29.851814 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.851758    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:29.851814 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.851774    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:29.851814 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.851803    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-64.ec2.internal"
Jul 29 20:47:29.861802 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:29.861788    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-64.ec2.internal"
Jul 29 20:47:29.861897 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:29.861809    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-64.ec2.internal\": node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.288347 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.288330    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.350698 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.349941    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"]
Jul 29 20:47:31.350860 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.350756    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:31.351748 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.351729    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:31.351846 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.351765    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:31.351846 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.351779    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:31.353192 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.353176    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:31.353365 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.353350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.353431 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.353379    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:31.353928 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.353912    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:31.353992 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.353942    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:31.353992 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.353953    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:31.354080 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.353912    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:31.354080 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.354022    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:31.354080 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.354036    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:31.355364 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.355350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.355407 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.355375    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:31.356019 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.356004    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:31.356083 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.356035    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:31.356083 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.356049    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:31.369828 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.369811    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-64.ec2.internal\" not found" node="ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.374208 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.374193    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-64.ec2.internal\" not found" node="ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.379385 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.379366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c12923416233decebd9e8669a4e889f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"0c12923416233decebd9e8669a4e889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.379438 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.379389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e75f6cc257a11d76b0facbb4329f9f9e-config\") pod \"kube-apiserver-proxy-ip-10-0-136-64.ec2.internal\" (UID: \"e75f6cc257a11d76b0facbb4329f9f9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.379438 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.379406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c12923416233decebd9e8669a4e889f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"0c12923416233decebd9e8669a4e889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.388890 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.388875    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.480003 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.479976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c12923416233decebd9e8669a4e889f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"0c12923416233decebd9e8669a4e889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.480100 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.480005    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c12923416233decebd9e8669a4e889f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"0c12923416233decebd9e8669a4e889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.480100 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.480024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e75f6cc257a11d76b0facbb4329f9f9e-config\") pod \"kube-apiserver-proxy-ip-10-0-136-64.ec2.internal\" (UID: \"e75f6cc257a11d76b0facbb4329f9f9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.480100 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.480062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e75f6cc257a11d76b0facbb4329f9f9e-config\") pod \"kube-apiserver-proxy-ip-10-0-136-64.ec2.internal\" (UID: \"e75f6cc257a11d76b0facbb4329f9f9e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.480100 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.480071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0c12923416233decebd9e8669a4e889f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"0c12923416233decebd9e8669a4e889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.480100 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.480072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0c12923416233decebd9e8669a4e889f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal\" (UID: \"0c12923416233decebd9e8669a4e889f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.489747 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.489728    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.590261 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.590214    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.673432 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.673403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.679547 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:31.679533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:31.690483 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.690463    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.791046 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.791014    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.891529 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.891468    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:31.992151 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:31.992124    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:32.010285 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.010267    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 20:47:32.010423 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.010405    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 20:47:32.092206 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:32.092180    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:32.096886 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.096865    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:47:32.143851 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.143787    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 20:42:31 +0000 UTC" deadline="2028-04-22 13:58:55.389136142 +0000 UTC"
Jul 29 20:47:32.143851 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.143815    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15185h11m23.245324088s"
Jul 29 20:47:32.162725 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:32.162699    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode75f6cc257a11d76b0facbb4329f9f9e.slice/crio-4b4e6d59be201e5dd9265fe9f95e32763aaff8fe28ceda58fee5ed0a6581c1c9 WatchSource:0}: Error finding container 4b4e6d59be201e5dd9265fe9f95e32763aaff8fe28ceda58fee5ed0a6581c1c9: Status 404 returned error can't find the container with id 4b4e6d59be201e5dd9265fe9f95e32763aaff8fe28ceda58fee5ed0a6581c1c9
Jul 29 20:47:32.163133 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:32.163110    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c12923416233decebd9e8669a4e889f.slice/crio-4f67cc2103440a2338c46b538f4a1ec50bb4857c193478d68f96cf70e4690a14 WatchSource:0}: Error finding container 4f67cc2103440a2338c46b538f4a1ec50bb4857c193478d68f96cf70e4690a14: Status 404 returned error can't find the container with id 4f67cc2103440a2338c46b538f4a1ec50bb4857c193478d68f96cf70e4690a14
Jul 29 20:47:32.165000 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.164990    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 20:47:32.165951 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.165936    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 20:47:32.166020 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.165963    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 20:47:32.166020 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.165972    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeHasSufficientPID"
Jul 29 20:47:32.166962 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.166950    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:47:32.193029 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:32.193012    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:32.247346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.247326    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:47:32.252086 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.252047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" event={"ID":"0c12923416233decebd9e8669a4e889f","Type":"ContainerStarted","Data":"4f67cc2103440a2338c46b538f4a1ec50bb4857c193478d68f96cf70e4690a14"}
Jul 29 20:47:32.253794 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.253776    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal" event={"ID":"e75f6cc257a11d76b0facbb4329f9f9e","Type":"ContainerStarted","Data":"4b4e6d59be201e5dd9265fe9f95e32763aaff8fe28ceda58fee5ed0a6581c1c9"}
Jul 29 20:47:32.293437 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:32.293417    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:32.394015 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:32.393954    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:32.494497 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:32.494471    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-64.ec2.internal\" not found"
Jul 29 20:47:32.516352 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.516328    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:47:32.576466 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.576442    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:32.584555 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.584532    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 20:47:32.585516 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.585504    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal"
Jul 29 20:47:32.595163 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:32.595145    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 20:47:33.055499 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.055420    2579 apiserver.go:52] "Watching apiserver"
Jul 29 20:47:33.067119 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.067072    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 20:47:33.069079 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.069052    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal","openshift-network-diagnostics/network-check-target-hs6c5","openshift-network-operator/iptables-alerter-4s469","openshift-ovn-kubernetes/ovnkube-node-fvcdx","kube-system/global-pull-secret-syncer-6xrsn","kube-system/konnectivity-agent-cxrmk","kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k","openshift-dns/node-resolver-m4kxx","openshift-multus/multus-2mrqp","openshift-multus/multus-additional-cni-plugins-782xm","openshift-multus/network-metrics-daemon-5nmdf","openshift-cluster-node-tuning-operator/tuned-v4fsr","openshift-image-registry/node-ca-s9brg"]
Jul 29 20:47:33.071062 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.071039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.072568 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.072544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.073206 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.073180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 20:47:33.073616 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.073422    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 20:47:33.073616 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.073444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-trjrn\""
Jul 29 20:47:33.073616 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.073462    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 20:47:33.073616 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.073421    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 20:47:33.073920 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.073904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.074462 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.074370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:47:33.074699 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.074563    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pq97p\""
Jul 29 20:47:33.074699 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.074637    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 20:47:33.075726 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.075703    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.076633    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.076705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.076814    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.076936    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.076997    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hbw52\""
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.077386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.077457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:33.077864 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.077706    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 20:47:33.078457 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.078439    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 20:47:33.078542 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.078499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 20:47:33.078542 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.078509    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 20:47:33.078693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.078661    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 20:47:33.078889 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.078859    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kq7lr\""
Jul 29 20:47:33.078889 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.078872    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 20:47:33.080035 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.079570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.081710 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.081312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.082034 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.082016    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 20:47:33.082223 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.082208    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 20:47:33.082586 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.082439    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-76l8q\""
Jul 29 20:47:33.082795 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.082775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.083512 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.083481    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 20:47:33.084008 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.083805    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p28r9\""
Jul 29 20:47:33.084008 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.083886    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 20:47:33.084384 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.084358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:33.084474 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.084429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:33.085545 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.085412    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 20:47:33.085917 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.085892    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 20:47:33.086010 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.085941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 20:47:33.086092 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.086078    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.086206 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.086097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2ljhc\""
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.087728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.087874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysctl-d\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.087902    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-cni-netd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.087926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-cni-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.087948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-cni-multus\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.087977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-var-lib-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-env-overrides\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-system-cni-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-etc-kubernetes\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-modprobe-d\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysconfig\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.088143 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-run\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-lib-modules\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d3371a2-2439-4dde-a4c3-f06e1efc215d-iptables-alerter-script\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-slash\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-log-socket\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-ovnkube-config\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887zd\" (UniqueName: \"kubernetes.io/projected/359d2897-520f-4819-be13-0aee3f7896ad-kube-api-access-887zd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tdm7v\""
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-kubernetes\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-systemd\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-conf-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d3371a2-2439-4dde-a4c3-f06e1efc215d-host-slash\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-systemd-units\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088562    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-ovn\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.088783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a29fefac-1560-4a99-a379-73e0d857393e-etc-tuned\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c896j\" (UniqueName: \"kubernetes.io/projected/a29fefac-1560-4a99-a379-73e0d857393e-kube-api-access-c896j\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088631    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088630    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-netns\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mv86\" (UniqueName: \"kubernetes.io/projected/efdf74bf-a4a6-4b85-9aa6-39e1db215820-kube-api-access-2mv86\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysctl-conf\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-var-lib-kubelet\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-systemd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/359d2897-520f-4819-be13-0aee3f7896ad-ovn-node-metrics-cert\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efdf74bf-a4a6-4b85-9aa6-39e1db215820-cni-binary-copy\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-daemon-config\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-sys\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a29fefac-1560-4a99-a379-73e0d857393e-tmp\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-dbus\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-cnibin\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.088960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-k8s-cni-cncf-io\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.089667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-node-log\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-ovnkube-script-lib\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-host\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-etc-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-kubelet-config\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-kubelet\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55n9s\" (UniqueName: \"kubernetes.io/projected/3d3371a2-2439-4dde-a4c3-f06e1efc215d-kube-api-access-55n9s\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-os-release\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-run-netns\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-cni-bin\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-hostroot\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-kubelet\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089347    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-cni-bin\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-socket-dir-parent\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-multus-certs\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.090535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.089907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 20:47:33.091819 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.091799    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:33.091915 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.091805    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 20:47:33.092006 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.091903    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:33.092006 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.091970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zvfsr\""
Jul 29 20:47:33.094484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.094428    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 20:47:33.160641 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.160617    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dlc6n"
Jul 29 20:47:33.168482 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.168450    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dlc6n"
Jul 29 20:47:33.178789 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.178693    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 20:47:33.189554 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-ovnkube-script-lib\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.189670 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-host\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.189670 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-etc-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.189670 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-kubelet-config\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.189670 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-cni-binary-copy\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189711    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-host\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-kubelet-config\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-kubelet\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-etc-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189787    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55n9s\" (UniqueName: \"kubernetes.io/projected/3d3371a2-2439-4dde-a4c3-f06e1efc215d-kube-api-access-55n9s\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-kubelet\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.189882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lksww\" (UniqueName: \"kubernetes.io/projected/7e0f2dcf-741a-44e4-9255-331e08203b12-kube-api-access-lksww\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.189989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cxqn\" (UniqueName: \"kubernetes.io/projected/f2728239-662b-4632-acf7-8ed1b5287ca4-kube-api-access-7cxqn\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-os-release\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-run-netns\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-run-netns\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-os-release\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190164    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-cni-bin\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-hostroot\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-ovnkube-script-lib\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-kubelet\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190233    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-cni-bin\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-cni-bin\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-socket-dir-parent\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-kubelet\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-cni-bin\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-hostroot\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-multus-certs\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-multus-certs\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysctl-d\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-cni-netd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-socket-dir-parent\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16342c1c-a4d7-4067-8d0e-a53b55d46078-konnectivity-ca\") pod \"konnectivity-agent-cxrmk\" (UID: \"16342c1c-a4d7-4067-8d0e-a53b55d46078\") " pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-cni-netd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-cni-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysctl-d\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-os-release\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.190987 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190592    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-tmp-dir\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-cni-multus\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-var-lib-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190672    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-cni-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-system-cni-dir\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-var-lib-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-var-lib-cni-multus\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-cnibin\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a54988dd-54c1-44d6-ac26-8e9964f9794a-host\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-env-overrides\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-system-cni-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-etc-kubernetes\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-modprobe-d\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190912    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysconfig\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-run\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-lib-modules\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.191642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.190995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d3371a2-2439-4dde-a4c3-f06e1efc215d-iptables-alerter-script\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191059    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysconfig\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-slash\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-modprobe-d\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-log-socket\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-system-cni-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-slash\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-etc-kubernetes\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-run\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-log-socket\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191241    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-lib-modules\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-env-overrides\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-ovnkube-config\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887zd\" (UniqueName: \"kubernetes.io/projected/359d2897-520f-4819-be13-0aee3f7896ad-kube-api-access-887zd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-kubernetes\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.192484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-systemd\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16342c1c-a4d7-4067-8d0e-a53b55d46078-agent-certs\") pod \"konnectivity-agent-cxrmk\" (UID: \"16342c1c-a4d7-4067-8d0e-a53b55d46078\") " pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-hosts-file\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3d3371a2-2439-4dde-a4c3-f06e1efc215d-iptables-alerter-script\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b9l\" (UniqueName: \"kubernetes.io/projected/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-kube-api-access-t8b9l\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vt9l\" (UniqueName: \"kubernetes.io/projected/a54988dd-54c1-44d6-ac26-8e9964f9794a-kube-api-access-7vt9l\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-conf-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d3371a2-2439-4dde-a4c3-f06e1efc215d-host-slash\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.191718    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-kubernetes\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/359d2897-520f-4819-be13-0aee3f7896ad-ovnkube-config\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.191811    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret podName:4cb7aa5f-475b-4da5-b00a-1410e77e5f66 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:33.691780034 +0000 UTC m=+3.008268927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret") pod "global-pull-secret-syncer-6xrsn" (UID: "4cb7aa5f-475b-4da5-b00a-1410e77e5f66") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-conf-dir\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-systemd\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3d3371a2-2439-4dde-a4c3-f06e1efc215d-host-slash\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.193369 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-systemd-units\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-ovn\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-systemd-units\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.191980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-ovn\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nchsc\" (UniqueName: \"kubernetes.io/projected/205e2614-4cd5-45f8-9d05-42e122d96f6d-kube-api-access-nchsc\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a29fefac-1560-4a99-a379-73e0d857393e-etc-tuned\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c896j\" (UniqueName: \"kubernetes.io/projected/a29fefac-1560-4a99-a379-73e0d857393e-kube-api-access-c896j\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-netns\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mv86\" (UniqueName: \"kubernetes.io/projected/efdf74bf-a4a6-4b85-9aa6-39e1db215820-kube-api-access-2mv86\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysctl-conf\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-var-lib-kubelet\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-systemd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/359d2897-520f-4819-be13-0aee3f7896ad-ovn-node-metrics-cert\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-device-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efdf74bf-a4a6-4b85-9aa6-39e1db215820-cni-binary-copy\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-daemon-config\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-sys\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192357    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a29fefac-1560-4a99-a379-73e0d857393e-tmp\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-sys\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-netns\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-dbus\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-openvswitch\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-cnibin\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-run-systemd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192563    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-etc-sysctl-conf\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a29fefac-1560-4a99-a379-73e0d857393e-var-lib-kubelet\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-dbus\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192934    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/efdf74bf-a4a6-4b85-9aa6-39e1db215820-multus-daemon-config\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192467    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-cnibin\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.192985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-k8s-cni-cncf-io\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.193007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-node-log\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.194906 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.193030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a54988dd-54c1-44d6-ac26-8e9964f9794a-serviceca\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.195658 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.193049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.195658 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.193076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/efdf74bf-a4a6-4b85-9aa6-39e1db215820-cni-binary-copy\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.195658 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.193100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/359d2897-520f-4819-be13-0aee3f7896ad-node-log\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.195658 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.193136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/efdf74bf-a4a6-4b85-9aa6-39e1db215820-host-run-k8s-cni-cncf-io\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.196400 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.196072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a29fefac-1560-4a99-a379-73e0d857393e-etc-tuned\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.196400 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.196109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a29fefac-1560-4a99-a379-73e0d857393e-tmp\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.197075 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.197035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/359d2897-520f-4819-be13-0aee3f7896ad-ovn-node-metrics-cert\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.198392 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.198344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55n9s\" (UniqueName: \"kubernetes.io/projected/3d3371a2-2439-4dde-a4c3-f06e1efc215d-kube-api-access-55n9s\") pod \"iptables-alerter-4s469\" (UID: \"3d3371a2-2439-4dde-a4c3-f06e1efc215d\") " pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.203076 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.203051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-887zd\" (UniqueName: \"kubernetes.io/projected/359d2897-520f-4819-be13-0aee3f7896ad-kube-api-access-887zd\") pod \"ovnkube-node-fvcdx\" (UID: \"359d2897-520f-4819-be13-0aee3f7896ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.204205 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.203957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c896j\" (UniqueName: \"kubernetes.io/projected/a29fefac-1560-4a99-a379-73e0d857393e-kube-api-access-c896j\") pod \"tuned-v4fsr\" (UID: \"a29fefac-1560-4a99-a379-73e0d857393e\") " pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.204402 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.204383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mv86\" (UniqueName: \"kubernetes.io/projected/efdf74bf-a4a6-4b85-9aa6-39e1db215820-kube-api-access-2mv86\") pod \"multus-2mrqp\" (UID: \"efdf74bf-a4a6-4b85-9aa6-39e1db215820\") " pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.293971 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.293934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16342c1c-a4d7-4067-8d0e-a53b55d46078-konnectivity-ca\") pod \"konnectivity-agent-cxrmk\" (UID: \"16342c1c-a4d7-4067-8d0e-a53b55d46078\") " pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.293971 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.293973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.293990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-os-release\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-os-release\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-socket-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-tmp-dir\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-system-cni-dir\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-cnibin\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294212 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a54988dd-54c1-44d6-ac26-8e9964f9794a-host\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-system-cni-dir\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-cnibin\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16342c1c-a4d7-4067-8d0e-a53b55d46078-agent-certs\") pod \"konnectivity-agent-cxrmk\" (UID: \"16342c1c-a4d7-4067-8d0e-a53b55d46078\") " pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-registration-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-hosts-file\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b9l\" (UniqueName: \"kubernetes.io/projected/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-kube-api-access-t8b9l\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vt9l\" (UniqueName: \"kubernetes.io/projected/a54988dd-54c1-44d6-ac26-8e9964f9794a-kube-api-access-7vt9l\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294515    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nchsc\" (UniqueName: \"kubernetes.io/projected/205e2614-4cd5-45f8-9d05-42e122d96f6d-kube-api-access-nchsc\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-tmp-dir\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.294621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-device-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16342c1c-a4d7-4067-8d0e-a53b55d46078-konnectivity-ca\") pod \"konnectivity-agent-cxrmk\" (UID: \"16342c1c-a4d7-4067-8d0e-a53b55d46078\") " pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-device-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a54988dd-54c1-44d6-ac26-8e9964f9794a-host\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a54988dd-54c1-44d6-ac26-8e9964f9794a-serviceca\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-sys-fs\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-cni-binary-copy\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-hosts-file\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lksww\" (UniqueName: \"kubernetes.io/projected/7e0f2dcf-741a-44e4-9255-331e08203b12-kube-api-access-lksww\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/205e2614-4cd5-45f8-9d05-42e122d96f6d-etc-selinux\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.294893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cxqn\" (UniqueName: \"kubernetes.io/projected/f2728239-662b-4632-acf7-8ed1b5287ca4-kube-api-access-7cxqn\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.294946    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:33.295347 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.295009    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:33.794989967 +0000 UTC m=+3.111478874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:33.296015 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.295034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f2728239-662b-4632-acf7-8ed1b5287ca4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.296015 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.295193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a54988dd-54c1-44d6-ac26-8e9964f9794a-serviceca\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.296015 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.295377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-cni-binary-copy\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.296015 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.295665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f2728239-662b-4632-acf7-8ed1b5287ca4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.297257 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.297236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16342c1c-a4d7-4067-8d0e-a53b55d46078-agent-certs\") pod \"konnectivity-agent-cxrmk\" (UID: \"16342c1c-a4d7-4067-8d0e-a53b55d46078\") " pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.301152 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.301131    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:47:33.301262 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.301160    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:47:33.301262 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.301174    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhs42 for pod openshift-network-diagnostics/network-check-target-hs6c5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:33.301262 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.301251    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42 podName:8249ba4a-4c89-439c-b5bd-542ce56c45b6 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:33.801232141 +0000 UTC m=+3.117721051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zhs42" (UniqueName: "kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42") pod "network-check-target-hs6c5" (UID: "8249ba4a-4c89-439c-b5bd-542ce56c45b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:33.303020 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.302996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nchsc\" (UniqueName: \"kubernetes.io/projected/205e2614-4cd5-45f8-9d05-42e122d96f6d-kube-api-access-nchsc\") pod \"aws-ebs-csi-driver-node-5tz9k\" (UID: \"205e2614-4cd5-45f8-9d05-42e122d96f6d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.303241 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.303221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vt9l\" (UniqueName: \"kubernetes.io/projected/a54988dd-54c1-44d6-ac26-8e9964f9794a-kube-api-access-7vt9l\") pod \"node-ca-s9brg\" (UID: \"a54988dd-54c1-44d6-ac26-8e9964f9794a\") " pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.303763 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.303745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b9l\" (UniqueName: \"kubernetes.io/projected/33fdf5c4-3b9d-4f59-92e4-b66c57f165ac-kube-api-access-t8b9l\") pod \"node-resolver-m4kxx\" (UID: \"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac\") " pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.304257 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.304238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cxqn\" (UniqueName: \"kubernetes.io/projected/f2728239-662b-4632-acf7-8ed1b5287ca4-kube-api-access-7cxqn\") pod \"multus-additional-cni-plugins-782xm\" (UID: \"f2728239-662b-4632-acf7-8ed1b5287ca4\") " pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.304513 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.304495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lksww\" (UniqueName: \"kubernetes.io/projected/7e0f2dcf-741a-44e4-9255-331e08203b12-kube-api-access-lksww\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:33.390501 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.390419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2mrqp"
Jul 29 20:47:33.400698 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.400674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4fsr"
Jul 29 20:47:33.411663 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.411643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4s469"
Jul 29 20:47:33.417629 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.417610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:33.425506 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.425478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:33.433383 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.433359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m4kxx"
Jul 29 20:47:33.442264 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.442244    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-s9brg"
Jul 29 20:47:33.450053 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.450031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k"
Jul 29 20:47:33.456977 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.456960    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-782xm"
Jul 29 20:47:33.479083 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.479057    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 20:47:33.697836 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.697742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:33.697983 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.697917    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:33.697983 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.697982    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret podName:4cb7aa5f-475b-4da5-b00a-1410e77e5f66 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:34.697966335 +0000 UTC m=+4.014455227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret") pod "global-pull-secret-syncer-6xrsn" (UID: "4cb7aa5f-475b-4da5-b00a-1410e77e5f66") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:33.798471 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.798441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:33.798631 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.798590    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:33.798685 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.798658    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:34.798643101 +0000 UTC m=+4.115131994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:33.899418 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:33.899383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:33.899604 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.899580    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:47:33.899662 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.899615    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:47:33.899662 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.899630    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhs42 for pod openshift-network-diagnostics/network-check-target-hs6c5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:33.899721 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:33.899695    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42 podName:8249ba4a-4c89-439c-b5bd-542ce56c45b6 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:34.899674555 +0000 UTC m=+4.216163456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhs42" (UniqueName: "kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42") pod "network-check-target-hs6c5" (UID: "8249ba4a-4c89-439c-b5bd-542ce56c45b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:34.128687 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.128658    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359d2897_520f_4819_be13_0aee3f7896ad.slice/crio-03f054babfd417b5d478ac89930ecb989d4882b72d57176ff5d6fd6ed7a3fb40 WatchSource:0}: Error finding container 03f054babfd417b5d478ac89930ecb989d4882b72d57176ff5d6fd6ed7a3fb40: Status 404 returned error can't find the container with id 03f054babfd417b5d478ac89930ecb989d4882b72d57176ff5d6fd6ed7a3fb40
Jul 29 20:47:34.146890 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.146644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29fefac_1560_4a99_a379_73e0d857393e.slice/crio-08e7436c75980a09b0d0696f9a8b79c0ef14077ee4027e9ca8c4dcea05270ef6 WatchSource:0}: Error finding container 08e7436c75980a09b0d0696f9a8b79c0ef14077ee4027e9ca8c4dcea05270ef6: Status 404 returned error can't find the container with id 08e7436c75980a09b0d0696f9a8b79c0ef14077ee4027e9ca8c4dcea05270ef6
Jul 29 20:47:34.148676 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.148651    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205e2614_4cd5_45f8_9d05_42e122d96f6d.slice/crio-917c2851a17bf6358030cd42b377b7e518cace580152d8bacec311f71deacb8a WatchSource:0}: Error finding container 917c2851a17bf6358030cd42b377b7e518cace580152d8bacec311f71deacb8a: Status 404 returned error can't find the container with id 917c2851a17bf6358030cd42b377b7e518cace580152d8bacec311f71deacb8a
Jul 29 20:47:34.149345 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.149324    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54988dd_54c1_44d6_ac26_8e9964f9794a.slice/crio-0fc09be971eca09b137bd3907c3a347d59cb682ed67a9d348b9ea78eb35f07a6 WatchSource:0}: Error finding container 0fc09be971eca09b137bd3907c3a347d59cb682ed67a9d348b9ea78eb35f07a6: Status 404 returned error can't find the container with id 0fc09be971eca09b137bd3907c3a347d59cb682ed67a9d348b9ea78eb35f07a6
Jul 29 20:47:34.150856 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.150687    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16342c1c_a4d7_4067_8d0e_a53b55d46078.slice/crio-2a107be1601ea311c1fe7bb7191dcfe509e1ccb3e29e9d45793da3753be5e845 WatchSource:0}: Error finding container 2a107be1601ea311c1fe7bb7191dcfe509e1ccb3e29e9d45793da3753be5e845: Status 404 returned error can't find the container with id 2a107be1601ea311c1fe7bb7191dcfe509e1ccb3e29e9d45793da3753be5e845
Jul 29 20:47:34.151794 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.151770    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefdf74bf_a4a6_4b85_9aa6_39e1db215820.slice/crio-feb1da42ffa20cdf57e71da7013fb80c1f7b6a3446c7a81495c155981c36e845 WatchSource:0}: Error finding container feb1da42ffa20cdf57e71da7013fb80c1f7b6a3446c7a81495c155981c36e845: Status 404 returned error can't find the container with id feb1da42ffa20cdf57e71da7013fb80c1f7b6a3446c7a81495c155981c36e845
Jul 29 20:47:34.152550 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.152532    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fdf5c4_3b9d_4f59_92e4_b66c57f165ac.slice/crio-efbd077146e0e2cb980542723286e21fba4f98b454a917fc2caa1d1fb9c3c7cb WatchSource:0}: Error finding container efbd077146e0e2cb980542723286e21fba4f98b454a917fc2caa1d1fb9c3c7cb: Status 404 returned error can't find the container with id efbd077146e0e2cb980542723286e21fba4f98b454a917fc2caa1d1fb9c3c7cb
Jul 29 20:47:34.153681 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.153657    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2728239_662b_4632_acf7_8ed1b5287ca4.slice/crio-53ba342a9da631b216514e7d452f3886fb1d50073380759e0782d4be55d97ce2 WatchSource:0}: Error finding container 53ba342a9da631b216514e7d452f3886fb1d50073380759e0782d4be55d97ce2: Status 404 returned error can't find the container with id 53ba342a9da631b216514e7d452f3886fb1d50073380759e0782d4be55d97ce2
Jul 29 20:47:34.155064 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:47:34.154933    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3371a2_2439_4dde_a4c3_f06e1efc215d.slice/crio-cdd69b0786082ef166ff1cf9d268aca1a2c0cc812809d85ee1f9ec8cb21c21b0 WatchSource:0}: Error finding container cdd69b0786082ef166ff1cf9d268aca1a2c0cc812809d85ee1f9ec8cb21c21b0: Status 404 returned error can't find the container with id cdd69b0786082ef166ff1cf9d268aca1a2c0cc812809d85ee1f9ec8cb21c21b0
Jul 29 20:47:34.171374 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.171352    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:42:33 +0000 UTC" deadline="2028-03-29 13:44:39.899981039 +0000 UTC"
Jul 29 20:47:34.171436 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.171375    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14608h57m5.728608898s"
Jul 29 20:47:34.250315 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.250173    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:34.250458 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.250437    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:34.259844 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.258455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4s469" event={"ID":"3d3371a2-2439-4dde-a4c3-f06e1efc215d","Type":"ContainerStarted","Data":"cdd69b0786082ef166ff1cf9d268aca1a2c0cc812809d85ee1f9ec8cb21c21b0"}
Jul 29 20:47:34.261177 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.261154    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m4kxx" event={"ID":"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac","Type":"ContainerStarted","Data":"efbd077146e0e2cb980542723286e21fba4f98b454a917fc2caa1d1fb9c3c7cb"}
Jul 29 20:47:34.262064 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.262045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9brg" event={"ID":"a54988dd-54c1-44d6-ac26-8e9964f9794a","Type":"ContainerStarted","Data":"0fc09be971eca09b137bd3907c3a347d59cb682ed67a9d348b9ea78eb35f07a6"}
Jul 29 20:47:34.263220 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.263201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"03f054babfd417b5d478ac89930ecb989d4882b72d57176ff5d6fd6ed7a3fb40"}
Jul 29 20:47:34.265036 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.265017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal" event={"ID":"e75f6cc257a11d76b0facbb4329f9f9e","Type":"ContainerStarted","Data":"909d12ad9c419b7ee0ae0193d5ce147d61d604967d0c36fe2bc8fa5727985edb"}
Jul 29 20:47:34.266160 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.266141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerStarted","Data":"53ba342a9da631b216514e7d452f3886fb1d50073380759e0782d4be55d97ce2"}
Jul 29 20:47:34.267237 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.267217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mrqp" event={"ID":"efdf74bf-a4a6-4b85-9aa6-39e1db215820","Type":"ContainerStarted","Data":"feb1da42ffa20cdf57e71da7013fb80c1f7b6a3446c7a81495c155981c36e845"}
Jul 29 20:47:34.268270 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.268249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cxrmk" event={"ID":"16342c1c-a4d7-4067-8d0e-a53b55d46078","Type":"ContainerStarted","Data":"2a107be1601ea311c1fe7bb7191dcfe509e1ccb3e29e9d45793da3753be5e845"}
Jul 29 20:47:34.269349 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.269314    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k" event={"ID":"205e2614-4cd5-45f8-9d05-42e122d96f6d","Type":"ContainerStarted","Data":"917c2851a17bf6358030cd42b377b7e518cace580152d8bacec311f71deacb8a"}
Jul 29 20:47:34.270389 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.270367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4fsr" event={"ID":"a29fefac-1560-4a99-a379-73e0d857393e","Type":"ContainerStarted","Data":"08e7436c75980a09b0d0696f9a8b79c0ef14077ee4027e9ca8c4dcea05270ef6"}
Jul 29 20:47:34.704910 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.704828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:34.705083 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.704993    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:34.705083 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.705056    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret podName:4cb7aa5f-475b-4da5-b00a-1410e77e5f66 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:36.705037173 +0000 UTC m=+6.021526073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret") pod "global-pull-secret-syncer-6xrsn" (UID: "4cb7aa5f-475b-4da5-b00a-1410e77e5f66") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:34.805354 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.805318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:34.805521 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.805485    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:34.805585 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.805548    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:36.80552992 +0000 UTC m=+6.122018827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:34.906154 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:34.906120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:34.906343 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.906299    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:47:34.906343 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.906323    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:47:34.906343 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.906337    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhs42 for pod openshift-network-diagnostics/network-check-target-hs6c5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:34.906516 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:34.906403    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42 podName:8249ba4a-4c89-439c-b5bd-542ce56c45b6 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:36.906384435 +0000 UTC m=+6.222873338 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhs42" (UniqueName: "kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42") pod "network-check-target-hs6c5" (UID: "8249ba4a-4c89-439c-b5bd-542ce56c45b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:35.172418 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:35.172272    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 20:42:33 +0000 UTC" deadline="2028-02-19 07:47:57.597192436 +0000 UTC"
Jul 29 20:47:35.172418 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:35.172343    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13667h0m22.4248534s"
Jul 29 20:47:35.256974 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:35.254847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:35.256974 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:35.254978    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:35.256974 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:35.255512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:35.256974 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:35.255609    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:35.292277 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:35.292107    2579 generic.go:356] "Generic (PLEG): container finished" podID="0c12923416233decebd9e8669a4e889f" containerID="84d5fc79cbc54cf83f422d3722fd68583ec03d219751e82ab17b4fd7d54f4185" exitCode=0
Jul 29 20:47:35.293169 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:35.293056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" event={"ID":"0c12923416233decebd9e8669a4e889f","Type":"ContainerDied","Data":"84d5fc79cbc54cf83f422d3722fd68583ec03d219751e82ab17b4fd7d54f4185"}
Jul 29 20:47:35.309705 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:35.309207    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-64.ec2.internal" podStartSLOduration=3.309187276 podStartE2EDuration="3.309187276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:34.280131708 +0000 UTC m=+3.596620625" watchObservedRunningTime="2026-07-29 20:47:35.309187276 +0000 UTC m=+4.625676193"
Jul 29 20:47:36.250404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:36.250371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:36.250880 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.250522    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:36.317642 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:36.317607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" event={"ID":"0c12923416233decebd9e8669a4e889f","Type":"ContainerStarted","Data":"8f55e7ebf8a95b232f5bb54873d4dfdd9c53538505308def181b141e9d9e9640"}
Jul 29 20:47:36.723554 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:36.723470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:36.723733 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.723714    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:36.723808 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.723782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret podName:4cb7aa5f-475b-4da5-b00a-1410e77e5f66 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:40.723763213 +0000 UTC m=+10.040252114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret") pod "global-pull-secret-syncer-6xrsn" (UID: "4cb7aa5f-475b-4da5-b00a-1410e77e5f66") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:36.823914 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:36.823874    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:36.824069 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.824033    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:36.824125 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.824100    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:40.824081658 +0000 UTC m=+10.140570557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:36.924748 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:36.924706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:36.925123 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.924968    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:47:36.925123 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.925002    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:47:36.925123 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.925016    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhs42 for pod openshift-network-diagnostics/network-check-target-hs6c5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:36.925123 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:36.925077    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42 podName:8249ba4a-4c89-439c-b5bd-542ce56c45b6 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:40.925058541 +0000 UTC m=+10.241547437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhs42" (UniqueName: "kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42") pod "network-check-target-hs6c5" (UID: "8249ba4a-4c89-439c-b5bd-542ce56c45b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:37.251198 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:37.250480    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:37.251198 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:37.250616    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:37.251198 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:37.250663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:37.251198 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:37.250776    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:38.250526 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:38.250492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:38.250711 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:38.250647    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:39.249613 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:39.249547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:39.249613 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:39.249593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:39.250093 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:39.249681    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:39.250093 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:39.249780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:40.250678 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:40.250559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:40.251140 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.250705    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:40.756090 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:40.755652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:40.756090 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.755818    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:40.756090 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.755885    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret podName:4cb7aa5f-475b-4da5-b00a-1410e77e5f66 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:48.755868278 +0000 UTC m=+18.072357184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret") pod "global-pull-secret-syncer-6xrsn" (UID: "4cb7aa5f-475b-4da5-b00a-1410e77e5f66") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:40.856301 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:40.856249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:40.856496 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.856454    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:40.856561 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.856527    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:48.856507092 +0000 UTC m=+18.172995987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:40.956754 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:40.956702    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:40.956938 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.956916    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:47:40.956938 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.956938    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:47:40.957058 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.956951    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhs42 for pod openshift-network-diagnostics/network-check-target-hs6c5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:40.957058 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:40.957013    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42 podName:8249ba4a-4c89-439c-b5bd-542ce56c45b6 nodeName:}" failed. No retries permitted until 2026-07-29 20:47:48.956995931 +0000 UTC m=+18.273484843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhs42" (UniqueName: "kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42") pod "network-check-target-hs6c5" (UID: "8249ba4a-4c89-439c-b5bd-542ce56c45b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:41.250511 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:41.250474    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:41.250672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:41.250590    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:41.251007 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:41.250816    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:41.251007 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:41.250922    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:42.250429 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:42.249914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:42.250429 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:42.250059    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:43.249781 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:43.249750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:43.250215 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:43.249793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:43.250215 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:43.249890    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:43.250215 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:43.249981    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:44.249667 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:44.249638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:44.249836 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:44.249752    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:45.249949 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:45.249911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:45.250426 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:45.249920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:45.250426 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:45.250041    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:45.250426 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:45.250117    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:46.249628 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:46.249588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:46.249812 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:46.249715    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:47.249608 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:47.249570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:47.250028 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:47.249621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:47.250028 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:47.249716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:47.250028 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:47.249835    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:48.250539 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:48.250504    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:48.251010 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:48.250625    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:48.813283 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:48.813249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:48.813499 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:48.813382    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:48.813499 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:48.813456    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret podName:4cb7aa5f-475b-4da5-b00a-1410e77e5f66 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:04.813434181 +0000 UTC m=+34.129923083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret") pod "global-pull-secret-syncer-6xrsn" (UID: "4cb7aa5f-475b-4da5-b00a-1410e77e5f66") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:47:48.913780 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:48.913749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:48.913943 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:48.913858    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:48.913943 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:48.913920    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:04.913902115 +0000 UTC m=+34.230391008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:47:49.015052 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:49.014978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:49.015186 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:49.015135    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:47:49.015186 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:49.015158    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:47:49.015186 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:49.015172    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhs42 for pod openshift-network-diagnostics/network-check-target-hs6c5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:49.015326 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:49.015218    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42 podName:8249ba4a-4c89-439c-b5bd-542ce56c45b6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:05.015204465 +0000 UTC m=+34.331693357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhs42" (UniqueName: "kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42") pod "network-check-target-hs6c5" (UID: "8249ba4a-4c89-439c-b5bd-542ce56c45b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:47:49.249706 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:49.249678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:49.249870 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:49.249781    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:49.249945 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:49.249862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:49.250006 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:49.249965    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:50.249604 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:50.249572    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:50.250067 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:50.249690    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:51.251491 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:51.251458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:51.251921 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:51.251572    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:51.251921 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:51.251634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:51.251921 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:51.251780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:52.249638 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:52.249605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:52.249822 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:52.249723    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:53.250963 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.250539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:53.251743 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.250608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:53.251743 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:53.251068    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:53.251743 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:53.251113    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:53.359256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.359210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"f2ca25df0d552d19c6520834fcb20140e52abf78d537d2f79ca22b21dbce0107"}
Jul 29 20:47:53.359460 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.359334    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"616ad37f7481d12f3e64887242bd14c4c39c0e2eaa6d642b6c0df5b5b89fab88"}
Jul 29 20:47:53.359460 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.359373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"40cf03ecb3ef75f60489418588d14c35c4fa7b5d1c519006d25ce045ba026c41"}
Jul 29 20:47:53.359460 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.359407    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"fa083c2260e84dced79aae007f1ec5f5ff3fe7bd18dc3ef260e908c32f032356"}
Jul 29 20:47:53.359460 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.359449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"d9fcb9495d634076756a8947d7f780dc9455bcc622834dd6e35970c618cf0942"}
Jul 29 20:47:53.359661 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.359495    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"27233f4d381b5ca770aaedc90eff2a64470dc04329a68fded026ba3c1fb1ac55"}
Jul 29 20:47:53.360892 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.360867    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2728239-662b-4632-acf7-8ed1b5287ca4" containerID="eb09af9b6fd85547126698e6cdab8552cdf1e858e90f51e46642cd9312829ccb" exitCode=0
Jul 29 20:47:53.361012 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.360937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerDied","Data":"eb09af9b6fd85547126698e6cdab8552cdf1e858e90f51e46642cd9312829ccb"}
Jul 29 20:47:53.362784 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.362733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2mrqp" event={"ID":"efdf74bf-a4a6-4b85-9aa6-39e1db215820","Type":"ContainerStarted","Data":"7030f035fe9ad8cf85c230351bd69c7968a2483674d1f6adb286429ea724154a"}
Jul 29 20:47:53.365084 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.365056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cxrmk" event={"ID":"16342c1c-a4d7-4067-8d0e-a53b55d46078","Type":"ContainerStarted","Data":"d8db58f8accf07a41cde0bb3daaa9d9894d3c9a9eda1b46ed573c7b056066c5c"}
Jul 29 20:47:53.369385 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.369360    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k" event={"ID":"205e2614-4cd5-45f8-9d05-42e122d96f6d","Type":"ContainerStarted","Data":"d9d3a0390ae3b5200edcf4c060666f72fd9305e617334419c98913fb1b61135e"}
Jul 29 20:47:53.370902 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.370879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4fsr" event={"ID":"a29fefac-1560-4a99-a379-73e0d857393e","Type":"ContainerStarted","Data":"04fe8a50e2eea9244661c1741835963d2cfdac4e34f4934645ad6e668688ab0f"}
Jul 29 20:47:53.372439 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.372415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m4kxx" event={"ID":"33fdf5c4-3b9d-4f59-92e4-b66c57f165ac","Type":"ContainerStarted","Data":"b379045705f6493710a859d59d8a8078dde18ec05c98331fd18579e5bc1bba3a"}
Jul 29 20:47:53.376795 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.376761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-s9brg" event={"ID":"a54988dd-54c1-44d6-ac26-8e9964f9794a","Type":"ContainerStarted","Data":"490d9fb0929691c4760a5ae47996197ff41fba49bc4a7ccbbac0d140ae615506"}
Jul 29 20:47:53.382776 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.382728    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-64.ec2.internal" podStartSLOduration=21.38271177 podStartE2EDuration="21.38271177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:36.333364783 +0000 UTC m=+5.649853696" watchObservedRunningTime="2026-07-29 20:47:53.38271177 +0000 UTC m=+22.699200687"
Jul 29 20:47:53.396373 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.396310    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m4kxx" podStartSLOduration=4.126846931 podStartE2EDuration="22.3962757s" totalImagesPullingTime="18.269428769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:53.395591128 +0000 UTC m=+22.712080045" watchObservedRunningTime="2026-07-29 20:47:53.3962757 +0000 UTC m=+22.712764617"
Jul 29 20:47:53.456122 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.456075    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-s9brg" podStartSLOduration=7.901965161 podStartE2EDuration="22.456059879s" totalImagesPullingTime="14.554094718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:53.425252423 +0000 UTC m=+22.741741338" watchObservedRunningTime="2026-07-29 20:47:53.456059879 +0000 UTC m=+22.772548796"
Jul 29 20:47:53.456585 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.456560    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2mrqp" podStartSLOduration=4.136220104 podStartE2EDuration="22.456551521s" totalImagesPullingTime="18.320331417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:53.455622056 +0000 UTC m=+22.772110972" watchObservedRunningTime="2026-07-29 20:47:53.456551521 +0000 UTC m=+22.773040436"
Jul 29 20:47:53.474409 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:53.474357    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v4fsr" podStartSLOduration=4.1890116729999995 podStartE2EDuration="22.474341033s" totalImagesPullingTime="18.28532936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:53.474125551 +0000 UTC m=+22.790614467" watchObservedRunningTime="2026-07-29 20:47:53.474341033 +0000 UTC m=+22.790829948"
Jul 29 20:47:54.001356 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.001328    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 20:47:54.167759 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.167595    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T20:47:54.001347555Z","UUID":"22b03f89-550f-4b7c-81f6-476ebd5ce348","Handler":null,"Name":"","Endpoint":""}
Jul 29 20:47:54.169200 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.169177    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 20:47:54.169344 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.169222    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 20:47:54.250298 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.250202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:54.250490 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:54.250371    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:54.381597 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.381562    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k" event={"ID":"205e2614-4cd5-45f8-9d05-42e122d96f6d","Type":"ContainerStarted","Data":"6234410f39ea0c670fc9af6e3151a65d26ba81c2698f8674fe080566127956e0"}
Jul 29 20:47:54.383236 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.383180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4s469" event={"ID":"3d3371a2-2439-4dde-a4c3-f06e1efc215d","Type":"ContainerStarted","Data":"79110a7129e0cc25214f67c6c20e12b32f2c4bbcb39319e9281f6ea5cb6e9c6f"}
Jul 29 20:47:54.396175 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.396120    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4s469" podStartSLOduration=5.126159811 podStartE2EDuration="23.3961047s" totalImagesPullingTime="18.269944889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:54.395866956 +0000 UTC m=+23.712355871" watchObservedRunningTime="2026-07-29 20:47:54.3961047 +0000 UTC m=+23.712593615"
Jul 29 20:47:54.396485 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:54.396438    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cxrmk" podStartSLOduration=8.843358913 podStartE2EDuration="23.396430063s" totalImagesPullingTime="14.55307115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:53.487842212 +0000 UTC m=+22.804331128" watchObservedRunningTime="2026-07-29 20:47:54.396430063 +0000 UTC m=+23.712918982"
Jul 29 20:47:55.250476 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:55.250445    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:55.250636 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:55.250575    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:55.250636 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:55.250621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:55.250760 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:55.250727    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:55.390031 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:55.389992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"90b285898a16d01f000ee9367d15ae4a149f40f7d1a861eeb6165b2cfcfcd656"}
Jul 29 20:47:56.249826 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:56.249791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:56.250019 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:56.249908    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:56.394845 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:56.394807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k" event={"ID":"205e2614-4cd5-45f8-9d05-42e122d96f6d","Type":"ContainerStarted","Data":"4c30d67d0a4c3b1ff77cd43a38f1ed7155631e72f8767371886af1158cebdf19"}
Jul 29 20:47:56.412386 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:56.412330    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5tz9k" podStartSLOduration=4.4647643949999996 podStartE2EDuration="25.412314681s" totalImagesPullingTime="20.947550286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:56.411552607 +0000 UTC m=+25.728041523" watchObservedRunningTime="2026-07-29 20:47:56.412314681 +0000 UTC m=+25.728803595"
Jul 29 20:47:56.423813 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:56.423791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:56.424344 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:56.424322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:57.249848 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:57.249817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:57.250008 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:57.249818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:57.250008 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:57.249952    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:57.250111 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:57.250019    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:57.398036 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:57.397827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:57.398427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:57.398312    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cxrmk"
Jul 29 20:47:58.250419 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:58.250389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:58.250601 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:58.250511    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:47:58.403091 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:58.403054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" event={"ID":"359d2897-520f-4819-be13-0aee3f7896ad","Type":"ContainerStarted","Data":"e3d7bfa8b64b03af6e90bfeaa4b047ca872210b731bdffaa9ccc4d676de2c367"}
Jul 29 20:47:58.403902 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:58.403330    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:58.404882 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:58.404854    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2728239-662b-4632-acf7-8ed1b5287ca4" containerID="0829faaf0e4b62ef0fde21acf356dc63b024aa513e4d580ab60a2d74136b99e6" exitCode=0
Jul 29 20:47:58.404983 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:58.404922    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerDied","Data":"0829faaf0e4b62ef0fde21acf356dc63b024aa513e4d580ab60a2d74136b99e6"}
Jul 29 20:47:58.418726 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:58.418707    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:58.428211 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:58.428174    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx" podStartSLOduration=9.075543205 podStartE2EDuration="27.428161315s" totalImagesPullingTime="18.35261811s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:47:58.427727439 +0000 UTC m=+27.744216391" watchObservedRunningTime="2026-07-29 20:47:58.428161315 +0000 UTC m=+27.744650257"
Jul 29 20:47:59.249817 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.249588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:59.250029 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.249646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:59.250029 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:59.249838    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:59.250029 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:59.249935    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:59.407312 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.407257    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:59.407727 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.407652    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:59.422380 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.422358    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:47:59.602528 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.602503    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xrsn"]
Jul 29 20:47:59.602647 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.602633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:47:59.602773 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:59.602746    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:47:59.611617 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.611598    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hs6c5"]
Jul 29 20:47:59.611703 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.611689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:47:59.611780 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:59.611763    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:47:59.612139 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.612120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5nmdf"]
Jul 29 20:47:59.612210 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:47:59.612204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:47:59.612300 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:47:59.612272    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:48:00.411234 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:00.411200    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2728239-662b-4632-acf7-8ed1b5287ca4" containerID="a83e42e0e7cfa4a42528b2805745888f85bb268ffaa512b6018b56f0a73f9367" exitCode=0
Jul 29 20:48:00.411682 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:00.411309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerDied","Data":"a83e42e0e7cfa4a42528b2805745888f85bb268ffaa512b6018b56f0a73f9367"}
Jul 29 20:48:01.250747 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:01.250712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:01.250923 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:01.250798    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:48:01.250923 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:01.250872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:48:01.251006 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:01.250963    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:48:01.251006 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:01.251003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:48:01.251072 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:01.251052    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:48:02.417418 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:02.417218    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2728239-662b-4632-acf7-8ed1b5287ca4" containerID="d329320ebf32b3ac7b03bf86e3488414c55d259ebbfc38611e548102816d4a6d" exitCode=0
Jul 29 20:48:02.417847 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:02.417313    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerDied","Data":"d329320ebf32b3ac7b03bf86e3488414c55d259ebbfc38611e548102816d4a6d"}
Jul 29 20:48:03.250305 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:03.250263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:03.250461 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:03.250262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:48:03.250461 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:03.250396    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hs6c5" podUID="8249ba4a-4c89-439c-b5bd-542ce56c45b6"
Jul 29 20:48:03.250558 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:03.250263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:48:03.250607 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:03.250479    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:48:03.250607 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:03.250566    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xrsn" podUID="4cb7aa5f-475b-4da5-b00a-1410e77e5f66"
Jul 29 20:48:04.831529 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:04.831491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:48:04.831975 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:04.831642    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 20:48:04.831975 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:04.831719    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret podName:4cb7aa5f-475b-4da5-b00a-1410e77e5f66 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:36.831698873 +0000 UTC m=+66.148187767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret") pod "global-pull-secret-syncer-6xrsn" (UID: "4cb7aa5f-475b-4da5-b00a-1410e77e5f66") : object "kube-system"/"original-pull-secret" not registered
Jul 29 20:48:04.932030 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:04.932000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:48:04.932198 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:04.932175    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:48:04.932276 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:04.932257    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:36.932234439 +0000 UTC m=+66.248723342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 20:48:05.033115 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.033079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:05.033304 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.033270    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 20:48:05.033372 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.033307    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 20:48:05.033372 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.033321    2579 projected.go:196] Error preparing data for projected volume kube-api-access-zhs42 for pod openshift-network-diagnostics/network-check-target-hs6c5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:48:05.033450 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.033383    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42 podName:8249ba4a-4c89-439c-b5bd-542ce56c45b6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:37.033365385 +0000 UTC m=+66.349854295 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zhs42" (UniqueName: "kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42") pod "network-check-target-hs6c5" (UID: "8249ba4a-4c89-439c-b5bd-542ce56c45b6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 20:48:05.042005 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.041986    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-64.ec2.internal" event="NodeReady"
Jul 29 20:48:05.042141 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.042125    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 20:48:05.077393 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.077370    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-575f7f84fb-2t52z"]
Jul 29 20:48:05.080322 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.080284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.082924 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.082692    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 20:48:05.082924 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.082826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 20:48:05.083086 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.082948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 20:48:05.083086 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.082948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ws2kf\""
Jul 29 20:48:05.092479 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.092459    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 20:48:05.098442 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.097719    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575f7f84fb-2t52z"]
Jul 29 20:48:05.098553 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.098492    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"]
Jul 29 20:48:05.100489 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.100469    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z9b6g"]
Jul 29 20:48:05.100607 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.100593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:05.102401 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.102383    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xk52g"]
Jul 29 20:48:05.102496 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.102478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.103055 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.102913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-245mr\""
Jul 29 20:48:05.103055 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.102926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 29 20:48:05.103055 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.102934    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 29 20:48:05.104584 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.104563    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 20:48:05.104584 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.104571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgjxr\""
Jul 29 20:48:05.104755 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.104676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.105036 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.105020    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 20:48:05.108348 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.108330    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 20:48:05.108635 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.108618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5tsgl\""
Jul 29 20:48:05.108883 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.108657    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 20:48:05.109272 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.109250    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"]
Jul 29 20:48:05.109403 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.109386    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 20:48:05.112366 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.112330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z9b6g"]
Jul 29 20:48:05.112887 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.112870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xk52g"]
Jul 29 20:48:05.234805 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.234767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b691b5e0-c6cc-479d-a23c-e9e96fc40efb-cert\") pod \"ingress-canary-xk52g\" (UID: \"b691b5e0-c6cc-479d-a23c-e9e96fc40efb\") " pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.234805 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.234807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.235095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.234829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-installation-pull-secrets\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.234969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:05.235095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fef84ca-71bc-4047-851d-0b83d1c25756-ca-trust-extracted\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c51b7261-61be-4338-a660-f6b2d192525d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:05.235095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-certificates\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-trusted-ca\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235097    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-bound-sa-token\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8528ab-0a79-4d17-807a-597976ead2f6-config-volume\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-image-registry-private-configuration\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cntz\" (UniqueName: \"kubernetes.io/projected/b691b5e0-c6cc-479d-a23c-e9e96fc40efb-kube-api-access-4cntz\") pod \"ingress-canary-xk52g\" (UID: \"b691b5e0-c6cc-479d-a23c-e9e96fc40efb\") " pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpfr\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-kube-api-access-pwpfr\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235278    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkjbz\" (UniqueName: \"kubernetes.io/projected/3b8528ab-0a79-4d17-807a-597976ead2f6-kube-api-access-gkjbz\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.235427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.235357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8528ab-0a79-4d17-807a-597976ead2f6-tmp-dir\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.249939 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.249913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:05.250077 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.249913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:48:05.250077 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.249914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:48:05.252897 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.252874    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 20:48:05.252998 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.252936    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 20:48:05.253333 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.253181    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffmvq\""
Jul 29 20:48:05.253333 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.253205    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vp87c\""
Jul 29 20:48:05.253333 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.253208    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 20:48:05.253333 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.253209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 20:48:05.336324 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8528ab-0a79-4d17-807a-597976ead2f6-tmp-dir\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.336324 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b691b5e0-c6cc-479d-a23c-e9e96fc40efb-cert\") pod \"ingress-canary-xk52g\" (UID: \"b691b5e0-c6cc-479d-a23c-e9e96fc40efb\") " pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.336519 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.336519 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-installation-pull-secrets\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.336519 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:05.336663 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.336518    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:48:05.336663 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fef84ca-71bc-4047-851d-0b83d1c25756-ca-trust-extracted\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.336663 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c51b7261-61be-4338-a660-f6b2d192525d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:05.336663 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.336587    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:05.836568561 +0000 UTC m=+35.153057465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:48:05.336663 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8528ab-0a79-4d17-807a-597976ead2f6-tmp-dir\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.336663 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.336620    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-certificates\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.336690    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:48:05.836671155 +0000 UTC m=+35.153160050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-trusted-ca\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-bound-sa-token\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8528ab-0a79-4d17-807a-597976ead2f6-config-volume\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-image-registry-private-configuration\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cntz\" (UniqueName: \"kubernetes.io/projected/b691b5e0-c6cc-479d-a23c-e9e96fc40efb-kube-api-access-4cntz\") pod \"ingress-canary-xk52g\" (UID: \"b691b5e0-c6cc-479d-a23c-e9e96fc40efb\") " pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.336968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.336861    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpfr\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-kube-api-access-pwpfr\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.337378 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.337201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fef84ca-71bc-4047-851d-0b83d1c25756-ca-trust-extracted\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.337378 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.337246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c51b7261-61be-4338-a660-f6b2d192525d-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:05.337378 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.337357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkjbz\" (UniqueName: \"kubernetes.io/projected/3b8528ab-0a79-4d17-807a-597976ead2f6-kube-api-access-gkjbz\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.337523 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.337389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b8528ab-0a79-4d17-807a-597976ead2f6-config-volume\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.337523 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.337393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.337523 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.337493    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:48:05.337523 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.337507    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:48:05.337703 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.337556    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:05.837542472 +0000 UTC m=+35.154031373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:48:05.337869 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.337830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-certificates\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.338239 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.338216    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-trusted-ca\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.341203 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.341180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b691b5e0-c6cc-479d-a23c-e9e96fc40efb-cert\") pod \"ingress-canary-xk52g\" (UID: \"b691b5e0-c6cc-479d-a23c-e9e96fc40efb\") " pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.341324 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.341199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-image-registry-private-configuration\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.341324 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.341202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-installation-pull-secrets\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.345382 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.345363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-bound-sa-token\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.345576 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.345557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkjbz\" (UniqueName: \"kubernetes.io/projected/3b8528ab-0a79-4d17-807a-597976ead2f6-kube-api-access-gkjbz\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.345677 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.345657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpfr\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-kube-api-access-pwpfr\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.346073 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.346054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cntz\" (UniqueName: \"kubernetes.io/projected/b691b5e0-c6cc-479d-a23c-e9e96fc40efb-kube-api-access-4cntz\") pod \"ingress-canary-xk52g\" (UID: \"b691b5e0-c6cc-479d-a23c-e9e96fc40efb\") " pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.427402 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.427377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xk52g"
Jul 29 20:48:05.840535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.840502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.840548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.840660    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.840664    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.840686    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.840717    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:06.840700343 +0000 UTC m=+36.157189250 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:05.840743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.840750    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:06.840732752 +0000 UTC m=+36.157221647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.840852    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:48:05.841159 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:05.840906    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:48:06.840891954 +0000 UTC m=+36.157380847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:48:06.583240 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:06.583041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xk52g"]
Jul 29 20:48:06.586675 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:48:06.586644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb691b5e0_c6cc_479d_a23c_e9e96fc40efb.slice/crio-4db1607790ec4dd46bc8a626924662e679aec1ae8affcea1a948a33d2015d225 WatchSource:0}: Error finding container 4db1607790ec4dd46bc8a626924662e679aec1ae8affcea1a948a33d2015d225: Status 404 returned error can't find the container with id 4db1607790ec4dd46bc8a626924662e679aec1ae8affcea1a948a33d2015d225
Jul 29 20:48:06.849266 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:06.849237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:06.849321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:06.849353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:06.849394    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:06.849450    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:48:08.849436102 +0000 UTC m=+38.165924998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:06.849467    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:06.849488    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:06.849522    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:06.849538    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:08.849525263 +0000 UTC m=+38.166014156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:48:06.849731 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:06.849569    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:08.849558375 +0000 UTC m=+38.166047273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:48:07.431615 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:07.431583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xk52g" event={"ID":"b691b5e0-c6cc-479d-a23c-e9e96fc40efb","Type":"ContainerStarted","Data":"4db1607790ec4dd46bc8a626924662e679aec1ae8affcea1a948a33d2015d225"}
Jul 29 20:48:07.434776 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:07.434750    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2728239-662b-4632-acf7-8ed1b5287ca4" containerID="ff8db1d10ab5708fe61325a43d7627296b492941c28776c806f197483fcbed02" exitCode=0
Jul 29 20:48:07.434894 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:07.434796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerDied","Data":"ff8db1d10ab5708fe61325a43d7627296b492941c28776c806f197483fcbed02"}
Jul 29 20:48:08.440496 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:08.440456    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2728239-662b-4632-acf7-8ed1b5287ca4" containerID="81a05ace12ea2abcfa02eee288e0f943879ad3cce1f41b015a7e0c5a552dc45c" exitCode=0
Jul 29 20:48:08.440921 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:08.440507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerDied","Data":"81a05ace12ea2abcfa02eee288e0f943879ad3cce1f41b015a7e0c5a552dc45c"}
Jul 29 20:48:08.865669 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:08.865625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:08.865873 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:08.865720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:08.865873 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:08.865758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:08.865873 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:08.865787    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:48:08.865873 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:08.865862    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:48:12.865842357 +0000 UTC m=+42.182331252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:48:08.865873 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:08.865867    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:48:08.866110 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:08.865887    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:48:08.866110 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:08.865943    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:12.865925486 +0000 UTC m=+42.182414404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:48:08.866110 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:08.865863    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:48:08.866110 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:08.865993    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:12.865978737 +0000 UTC m=+42.182467634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:48:09.446106 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:09.446077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-782xm" event={"ID":"f2728239-662b-4632-acf7-8ed1b5287ca4","Type":"ContainerStarted","Data":"6019ee128e4272d3c9b3edda405d288fac447122d65405ad34efc8c98fbb34e8"}
Jul 29 20:48:09.468972 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:09.468928    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-782xm" podStartSLOduration=9.418487077 podStartE2EDuration="38.468911278s" totalImagesPullingTime="29.050424201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:09.467180331 +0000 UTC m=+38.783669245" watchObservedRunningTime="2026-07-29 20:48:09.468911278 +0000 UTC m=+38.785400194"
Jul 29 20:48:10.450061 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:10.449897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xk52g" event={"ID":"b691b5e0-c6cc-479d-a23c-e9e96fc40efb","Type":"ContainerStarted","Data":"adfe0abffc0c914ea0faa693fe00cf325a2b7059c8831bfef2ae8c0e22467fe8"}
Jul 29 20:48:10.469871 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:10.469830    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xk52g" podStartSLOduration=2.676360564 podStartE2EDuration="5.469814275s" totalImagesPullingTime="2.793453711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:10.469211314 +0000 UTC m=+39.785700232" watchObservedRunningTime="2026-07-29 20:48:10.469814275 +0000 UTC m=+39.786303220"
Jul 29 20:48:12.898568 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:12.898533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:12.898593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:12.898624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:12.898702    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:12.898753    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:20.898741211 +0000 UTC m=+50.215230104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:12.898702    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:12.898822    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:48:20.898810606 +0000 UTC m=+50.215299501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:12.898779    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:12.898844    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:48:12.898928 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:12.898875    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:20.898865301 +0000 UTC m=+50.215354207 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:48:20.956604 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:20.956565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:20.956611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:20.956680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:20.956731    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:20.956748    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:20.956784    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:20.956806    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:20.956810    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:36.956789268 +0000 UTC m=+66.273278180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:20.956880    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:48:36.956866357 +0000 UTC m=+66.273355250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:48:20.957043 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:20.956892    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:48:36.956885035 +0000 UTC m=+66.273373928 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:48:31.431241 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:31.431205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvcdx"
Jul 29 20:48:36.861407 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.861362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:48:36.864041 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.864021    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 20:48:36.875546 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.875526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cb7aa5f-475b-4da5-b00a-1410e77e5f66-original-pull-secret\") pod \"global-pull-secret-syncer-6xrsn\" (UID: \"4cb7aa5f-475b-4da5-b00a-1410e77e5f66\") " pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:48:36.962264 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.962235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:48:36.962403 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.962278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:48:36.962403 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.962312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:48:36.962403 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.962331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:48:36.962552 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.962399    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:48:36.962552 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.962416    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:48:36.962552 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.962445    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:48:36.962552 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.962465    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:48:36.962552 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.962475    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:49:08.962454244 +0000 UTC m=+98.278943152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:48:36.962552 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.962495    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:49:08.96248545 +0000 UTC m=+98.278974343 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:48:36.962552 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.962509    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:49:08.962502196 +0000 UTC m=+98.278991090 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:48:36.964533 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:36.964516    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 20:48:36.972943 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.972926    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 20:48:36.972998 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:48:36.972969    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:49:40.972957068 +0000 UTC m=+130.289445961 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : secret "metrics-daemon-secret" not found
Jul 29 20:48:37.062971 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.062944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:37.065370 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.065353    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 20:48:37.073161 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.073144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xrsn"
Jul 29 20:48:37.075056 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.075040    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 20:48:37.086504 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.086483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhs42\" (UniqueName: \"kubernetes.io/projected/8249ba4a-4c89-439c-b5bd-542ce56c45b6-kube-api-access-zhs42\") pod \"network-check-target-hs6c5\" (UID: \"8249ba4a-4c89-439c-b5bd-542ce56c45b6\") " pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:37.193547 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.193515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xrsn"]
Jul 29 20:48:37.197903 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:48:37.197869    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb7aa5f_475b_4da5_b00a_1410e77e5f66.slice/crio-13098a446784611d68007be51bb9b01e6fe8f9a25886c921c90bb6008932bf8d WatchSource:0}: Error finding container 13098a446784611d68007be51bb9b01e6fe8f9a25886c921c90bb6008932bf8d: Status 404 returned error can't find the container with id 13098a446784611d68007be51bb9b01e6fe8f9a25886c921c90bb6008932bf8d
Jul 29 20:48:37.364283 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.364252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ffmvq\""
Jul 29 20:48:37.372216 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.372166    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:37.506278 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.506249    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hs6c5"]
Jul 29 20:48:37.510017 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:48:37.509994    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8249ba4a_4c89_439c_b5bd_542ce56c45b6.slice/crio-5f12fc6ce8e419df4877ae586662a194d978811c675ee1f98c41a29cc3dd70cf WatchSource:0}: Error finding container 5f12fc6ce8e419df4877ae586662a194d978811c675ee1f98c41a29cc3dd70cf: Status 404 returned error can't find the container with id 5f12fc6ce8e419df4877ae586662a194d978811c675ee1f98c41a29cc3dd70cf
Jul 29 20:48:37.512672 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:37.512649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6xrsn" event={"ID":"4cb7aa5f-475b-4da5-b00a-1410e77e5f66","Type":"ContainerStarted","Data":"13098a446784611d68007be51bb9b01e6fe8f9a25886c921c90bb6008932bf8d"}
Jul 29 20:48:38.518532 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:38.518494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hs6c5" event={"ID":"8249ba4a-4c89-439c-b5bd-542ce56c45b6","Type":"ContainerStarted","Data":"5f12fc6ce8e419df4877ae586662a194d978811c675ee1f98c41a29cc3dd70cf"}
Jul 29 20:48:42.529569 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:42.529526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6xrsn" event={"ID":"4cb7aa5f-475b-4da5-b00a-1410e77e5f66","Type":"ContainerStarted","Data":"4d7fbc45bd46c0a52abf39ab9a789fa496e0512d739904048d02ce87751c57c6"}
Jul 29 20:48:42.530967 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:42.530945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hs6c5" event={"ID":"8249ba4a-4c89-439c-b5bd-542ce56c45b6","Type":"ContainerStarted","Data":"a59ec480a0c2b94de85ad2743d3ac47569571f643519a2867c785f913ccf53a2"}
Jul 29 20:48:42.531072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:42.531002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:48:42.546865 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:42.546824    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6xrsn" podStartSLOduration=66.811113864 podStartE2EDuration="1m11.546813142s" totalImagesPullingTime="4.735699278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:42.546149699 +0000 UTC m=+71.862638614" watchObservedRunningTime="2026-07-29 20:48:42.546813142 +0000 UTC m=+71.863302056"
Jul 29 20:48:42.560943 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:48:42.560899    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hs6c5" podStartSLOduration=67.140537156 podStartE2EDuration="1m11.560886235s" totalImagesPullingTime="4.420349079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:48:42.56058102 +0000 UTC m=+71.877069935" watchObservedRunningTime="2026-07-29 20:48:42.560886235 +0000 UTC m=+71.877375150"
Jul 29 20:49:08.992072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:08.991935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:49:08.992072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:08.991980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:49:08.992072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:08.992026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:49:08.992672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:08.992102    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:49:08.992672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:08.992126    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:49:08.992672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:08.992148    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:49:08.992672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:08.992127    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:49:08.992672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:08.992209    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:50:12.992191923 +0000 UTC m=+162.308680819 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:49:08.992672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:08.992229    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:50:12.992219523 +0000 UTC m=+162.308708418 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:49:08.992672 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:08.992242    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:50:12.99223517 +0000 UTC m=+162.308724063 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:49:13.535189 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:13.535163    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hs6c5"
Jul 29 20:49:39.274904 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.274877    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"]
Jul 29 20:49:39.276883 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.276867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.281205 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.281178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 29 20:49:39.281205 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.281198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 20:49:39.281390 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.281180    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 20:49:39.281390 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.281255    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-79ph9\""
Jul 29 20:49:39.281390 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.281181    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 20:49:39.289388 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.289367    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"]
Jul 29 20:49:39.401026 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.400997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f348b2ed-b93b-4c40-a595-d361dcd24882-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674c485c6b-jmvvk\" (UID: \"f348b2ed-b93b-4c40-a595-d361dcd24882\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.401026 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.401028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b96nj\" (UniqueName: \"kubernetes.io/projected/f348b2ed-b93b-4c40-a595-d361dcd24882-kube-api-access-b96nj\") pod \"managed-serviceaccount-addon-agent-674c485c6b-jmvvk\" (UID: \"f348b2ed-b93b-4c40-a595-d361dcd24882\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.501934 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.501897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f348b2ed-b93b-4c40-a595-d361dcd24882-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674c485c6b-jmvvk\" (UID: \"f348b2ed-b93b-4c40-a595-d361dcd24882\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.501934 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.501936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b96nj\" (UniqueName: \"kubernetes.io/projected/f348b2ed-b93b-4c40-a595-d361dcd24882-kube-api-access-b96nj\") pod \"managed-serviceaccount-addon-agent-674c485c6b-jmvvk\" (UID: \"f348b2ed-b93b-4c40-a595-d361dcd24882\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.504106 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.504081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f348b2ed-b93b-4c40-a595-d361dcd24882-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-674c485c6b-jmvvk\" (UID: \"f348b2ed-b93b-4c40-a595-d361dcd24882\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.510165 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.510146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b96nj\" (UniqueName: \"kubernetes.io/projected/f348b2ed-b93b-4c40-a595-d361dcd24882-kube-api-access-b96nj\") pod \"managed-serviceaccount-addon-agent-674c485c6b-jmvvk\" (UID: \"f348b2ed-b93b-4c40-a595-d361dcd24882\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.603236 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.603214    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"
Jul 29 20:49:39.718086 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:39.718057    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk"]
Jul 29 20:49:39.721544 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:49:39.721514    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf348b2ed_b93b_4c40_a595_d361dcd24882.slice/crio-0dbd276bd6ee67c8698a13573482c8ce5ba2fd672d40b77cbd7be2cead9f75c4 WatchSource:0}: Error finding container 0dbd276bd6ee67c8698a13573482c8ce5ba2fd672d40b77cbd7be2cead9f75c4: Status 404 returned error can't find the container with id 0dbd276bd6ee67c8698a13573482c8ce5ba2fd672d40b77cbd7be2cead9f75c4
Jul 29 20:49:40.665686 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:40.665647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk" event={"ID":"f348b2ed-b93b-4c40-a595-d361dcd24882","Type":"ContainerStarted","Data":"0dbd276bd6ee67c8698a13573482c8ce5ba2fd672d40b77cbd7be2cead9f75c4"}
Jul 29 20:49:41.013600 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:41.013510    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:49:41.013773 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:41.013663    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 20:49:41.013773 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:49:41.013730    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:43.01370982 +0000 UTC m=+252.330198716 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : secret "metrics-daemon-secret" not found
Jul 29 20:49:42.671617 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:42.671581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk" event={"ID":"f348b2ed-b93b-4c40-a595-d361dcd24882","Type":"ContainerStarted","Data":"fc9e00db5836eced5e61cacbe1e5d90dcd8a1d2e8def2c63477e1a2ee4610ff0"}
Jul 29 20:49:42.687427 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:49:42.687370    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk" podStartSLOduration=1.124338878 podStartE2EDuration="3.687356622s" totalImagesPullingTime="2.563017744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:49:42.686793576 +0000 UTC m=+132.003282492" watchObservedRunningTime="2026-07-29 20:49:42.687356622 +0000 UTC m=+132.003845584"
Jul 29 20:50:08.093228 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:08.093188    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756"
Jul 29 20:50:08.112486 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:08.112453    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz" podUID="c51b7261-61be-4338-a660-f6b2d192525d"
Jul 29 20:50:08.120587 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:08.120562    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z9b6g" podUID="3b8528ab-0a79-4d17-807a-597976ead2f6"
Jul 29 20:50:08.267891 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:08.267854    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:50:08.732095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:50:08.732065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:50:08.732283 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:50:08.732065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:50:08.732283 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:50:08.732070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:50:13.042579 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:50:13.042531    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:50:13.042579 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:50:13.042593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:50:13.042639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:13.042684    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:13.042707    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575f7f84fb-2t52z: secret "image-registry-tls" not found
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:13.042746    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:13.042776    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls podName:2fef84ca-71bc-4047-851d-0b83d1c25756 nodeName:}" failed. No retries permitted until 2026-07-29 20:52:15.04275743 +0000 UTC m=+284.359246344 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls") pod "image-registry-575f7f84fb-2t52z" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756") : secret "image-registry-tls" not found
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:13.042751    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:13.042790    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls podName:3b8528ab-0a79-4d17-807a-597976ead2f6 nodeName:}" failed. No retries permitted until 2026-07-29 20:52:15.042784249 +0000 UTC m=+284.359273142 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls") pod "dns-default-z9b6g" (UID: "3b8528ab-0a79-4d17-807a-597976ead2f6") : secret "dns-default-metrics-tls" not found
Jul 29 20:50:13.043061 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:50:13.042810    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert podName:c51b7261-61be-4338-a660-f6b2d192525d nodeName:}" failed. No retries permitted until 2026-07-29 20:52:15.042798232 +0000 UTC m=+284.359287125 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-l8cbz" (UID: "c51b7261-61be-4338-a660-f6b2d192525d") : secret "networking-console-plugin-cert" not found
Jul 29 20:50:19.250169 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:50:19.250128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:51:39.629336 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.629304    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"]
Jul 29 20:51:39.631319 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.631304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.633748 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.633724    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 29 20:51:39.634539 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.634495    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 29 20:51:39.634539 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.634503    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 29 20:51:39.634725 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.634558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:51:39.634725 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.634558    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w8sdz\""
Jul 29 20:51:39.635761 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.635740    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-m7wjk"]
Jul 29 20:51:39.638064 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.637771    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"]
Jul 29 20:51:39.638064 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.637811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.639786 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.639764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.640396 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.640377    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 29 20:51:39.640481 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.640446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-gb5dm\""
Jul 29 20:51:39.640481 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.640453    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 29 20:51:39.640595 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.640515    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 20:51:39.640664 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.640648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 20:51:39.642361 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.642327    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 20:51:39.642970 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.642946    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 29 20:51:39.643256 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.643230    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 20:51:39.646041 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.646019    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 29 20:51:39.646312 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.646281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-crzbv\""
Jul 29 20:51:39.650253 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.650232    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 29 20:51:39.651124 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.651031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"]
Jul 29 20:51:39.652169 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.652145    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-m7wjk"]
Jul 29 20:51:39.652966 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.652947    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"]
Jul 29 20:51:39.659416 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwjw5\" (UniqueName: \"kubernetes.io/projected/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-kube-api-access-fwjw5\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.659522 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659434    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vn9k\" (UniqueName: \"kubernetes.io/projected/dd421a79-7fc7-46dc-9e78-3ac699f74657-kube-api-access-4vn9k\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.659522 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dd421a79-7fc7-46dc-9e78-3ac699f74657-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.659522 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.659654 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.659654 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-service-ca-bundle\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.659654 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-snapshots\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.659818 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-tmp\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.659818 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.659818 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-serving-cert\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.659818 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zzpz\" (UniqueName: \"kubernetes.io/projected/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-kube-api-access-4zzpz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.659989 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.659856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.760810 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.760775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vn9k\" (UniqueName: \"kubernetes.io/projected/dd421a79-7fc7-46dc-9e78-3ac699f74657-kube-api-access-4vn9k\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.761000 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.760826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dd421a79-7fc7-46dc-9e78-3ac699f74657-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.761000 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.760843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.761000 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.760862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.761000 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:39.760948    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:39.761216 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-service-ca-bundle\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761216 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:39.761025    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls podName:dd421a79-7fc7-46dc-9e78-3ac699f74657 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:40.261006063 +0000 UTC m=+249.577494964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pj425" (UID: "dd421a79-7fc7-46dc-9e78-3ac699f74657") : secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:39.761216 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-snapshots\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761216 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-tmp\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761216 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.761216 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-serving-cert\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761216 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761191    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zzpz\" (UniqueName: \"kubernetes.io/projected/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-kube-api-access-4zzpz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.761657 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761657 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwjw5\" (UniqueName: \"kubernetes.io/projected/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-kube-api-access-fwjw5\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761759 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-service-ca-bundle\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761759 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-snapshots\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.761866 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.761866 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/dd421a79-7fc7-46dc-9e78-3ac699f74657-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.761985 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.761962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-tmp\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.762342 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.762323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.763159 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.763137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.763457 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.763441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-serving-cert\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.769728 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.769700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zzpz\" (UniqueName: \"kubernetes.io/projected/c1034613-bed5-4ad8-908c-7aa2cdde9e1c-kube-api-access-4zzpz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2\" (UID: \"c1034613-bed5-4ad8-908c-7aa2cdde9e1c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.769850 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.769828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwjw5\" (UniqueName: \"kubernetes.io/projected/4ffc6d67-25c5-4c0b-9d7e-693c2a790be1-kube-api-access-fwjw5\") pod \"insights-operator-5885fdb759-m7wjk\" (UID: \"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1\") " pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:39.770135 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.770106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vn9k\" (UniqueName: \"kubernetes.io/projected/dd421a79-7fc7-46dc-9e78-3ac699f74657-kube-api-access-4vn9k\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:39.940522 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.940420    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"
Jul 29 20:51:39.950625 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:39.950599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-m7wjk"
Jul 29 20:51:40.067543 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:40.067508    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2"]
Jul 29 20:51:40.069958 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:51:40.069929    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1034613_bed5_4ad8_908c_7aa2cdde9e1c.slice/crio-0778cac8ed3f52a44bb0e52fd4e6c9b0711dd7da52a901c25a641f96257e9349 WatchSource:0}: Error finding container 0778cac8ed3f52a44bb0e52fd4e6c9b0711dd7da52a901c25a641f96257e9349: Status 404 returned error can't find the container with id 0778cac8ed3f52a44bb0e52fd4e6c9b0711dd7da52a901c25a641f96257e9349
Jul 29 20:51:40.085227 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:40.085204    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-m7wjk"]
Jul 29 20:51:40.088161 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:51:40.088137    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffc6d67_25c5_4c0b_9d7e_693c2a790be1.slice/crio-b809060fd9450b09701462dd6e4aa1b7b353345ff93f61bb629dbaa96bcec0c9 WatchSource:0}: Error finding container b809060fd9450b09701462dd6e4aa1b7b353345ff93f61bb629dbaa96bcec0c9: Status 404 returned error can't find the container with id b809060fd9450b09701462dd6e4aa1b7b353345ff93f61bb629dbaa96bcec0c9
Jul 29 20:51:40.264447 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:40.264342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:40.264710 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:40.264464    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:40.264710 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:40.264537    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls podName:dd421a79-7fc7-46dc-9e78-3ac699f74657 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:41.264517323 +0000 UTC m=+250.581006215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pj425" (UID: "dd421a79-7fc7-46dc-9e78-3ac699f74657") : secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:40.946629 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:40.946556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2" event={"ID":"c1034613-bed5-4ad8-908c-7aa2cdde9e1c","Type":"ContainerStarted","Data":"0778cac8ed3f52a44bb0e52fd4e6c9b0711dd7da52a901c25a641f96257e9349"}
Jul 29 20:51:40.948504 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:40.948337    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-m7wjk" event={"ID":"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1","Type":"ContainerStarted","Data":"b809060fd9450b09701462dd6e4aa1b7b353345ff93f61bb629dbaa96bcec0c9"}
Jul 29 20:51:41.272042 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:41.271960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:41.272210 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:41.272129    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:41.272284 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:41.272220    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls podName:dd421a79-7fc7-46dc-9e78-3ac699f74657 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:43.272198239 +0000 UTC m=+252.588687139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pj425" (UID: "dd421a79-7fc7-46dc-9e78-3ac699f74657") : secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:42.956188 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:42.956150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-m7wjk" event={"ID":"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1","Type":"ContainerStarted","Data":"cf23e31d3de995968d9acd1ec061a64064555fd01126b25594b498ac8d3de920"}
Jul 29 20:51:42.957486 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:42.957465    2579 generic.go:356] "Generic (PLEG): container finished" podID="f348b2ed-b93b-4c40-a595-d361dcd24882" containerID="fc9e00db5836eced5e61cacbe1e5d90dcd8a1d2e8def2c63477e1a2ee4610ff0" exitCode=255
Jul 29 20:51:42.957593 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:42.957544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk" event={"ID":"f348b2ed-b93b-4c40-a595-d361dcd24882","Type":"ContainerDied","Data":"fc9e00db5836eced5e61cacbe1e5d90dcd8a1d2e8def2c63477e1a2ee4610ff0"}
Jul 29 20:51:42.959037 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:42.958958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2" event={"ID":"c1034613-bed5-4ad8-908c-7aa2cdde9e1c","Type":"ContainerStarted","Data":"7faceed7dff5fc24c10800f642ad6b4502d62bf3859e1882c4e3096cb6f59f6f"}
Jul 29 20:51:42.963350 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:42.963336    2579 scope.go:117] "RemoveContainer" containerID="fc9e00db5836eced5e61cacbe1e5d90dcd8a1d2e8def2c63477e1a2ee4610ff0"
Jul 29 20:51:42.974182 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:42.974143    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-m7wjk" podStartSLOduration=1.5129735800000002 podStartE2EDuration="3.974126336s" totalImagesPullingTime="2.461152756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:42.973175808 +0000 UTC m=+252.289664728" watchObservedRunningTime="2026-07-29 20:51:42.974126336 +0000 UTC m=+252.290615252"
Jul 29 20:51:42.993091 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:42.993044    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2" podStartSLOduration=1.514584814 podStartE2EDuration="3.99302904s" totalImagesPullingTime="2.478444226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:42.991888914 +0000 UTC m=+252.308377830" watchObservedRunningTime="2026-07-29 20:51:42.99302904 +0000 UTC m=+252.309517956"
Jul 29 20:51:43.090228 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.090197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:51:43.090418 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.090355    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 29 20:51:43.090418 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.090414    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs podName:7e0f2dcf-741a-44e4-9255-331e08203b12 nodeName:}" failed. No retries permitted until 2026-07-29 20:53:45.090397612 +0000 UTC m=+374.406886506 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs") pod "network-metrics-daemon-5nmdf" (UID: "7e0f2dcf-741a-44e4-9255-331e08203b12") : secret "metrics-daemon-secret" not found
Jul 29 20:51:43.291952 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.291845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:43.292127 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.292115    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:43.292224 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.292187    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls podName:dd421a79-7fc7-46dc-9e78-3ac699f74657 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:47.292166283 +0000 UTC m=+256.608655243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pj425" (UID: "dd421a79-7fc7-46dc-9e78-3ac699f74657") : secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:43.393507 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.393474    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7pfpq"]
Jul 29 20:51:43.395852 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.395832    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.401874 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.401849    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 20:51:43.402019 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.401959    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 20:51:43.402145 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.402118    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9wqct\""
Jul 29 20:51:43.413284 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.413260    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7pfpq"]
Jul 29 20:51:43.493249 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.493222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd6703c2-da11-4da2-b67c-50513265ba18-crio-socket\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.493422 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.493256    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.493422 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.493329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.493422 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.493356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbn9p\" (UniqueName: \"kubernetes.io/projected/dd6703c2-da11-4da2-b67c-50513265ba18-kube-api-access-qbn9p\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.493422 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.493389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd6703c2-da11-4da2-b67c-50513265ba18-data-volume\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.594282 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.593968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.594282 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.594023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbn9p\" (UniqueName: \"kubernetes.io/projected/dd6703c2-da11-4da2-b67c-50513265ba18-kube-api-access-qbn9p\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.594282 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.594103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd6703c2-da11-4da2-b67c-50513265ba18-data-volume\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.594282 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.594166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd6703c2-da11-4da2-b67c-50513265ba18-crio-socket\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.594282 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.594194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.594627 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.594353    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:51:43.594627 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.594365    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:43.594627 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.594424    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:44.094405302 +0000 UTC m=+253.410894203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:43.594627 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:43.594452    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:44.094442085 +0000 UTC m=+253.410930998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : configmap "kube-rbac-proxy" not found
Jul 29 20:51:43.594627 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.594447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dd6703c2-da11-4da2-b67c-50513265ba18-crio-socket\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.594822 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.594763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd6703c2-da11-4da2-b67c-50513265ba18-data-volume\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.602081 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.602055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbn9p\" (UniqueName: \"kubernetes.io/projected/dd6703c2-da11-4da2-b67c-50513265ba18-kube-api-access-qbn9p\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:43.963372 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:43.963272    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-674c485c6b-jmvvk" event={"ID":"f348b2ed-b93b-4c40-a595-d361dcd24882","Type":"ContainerStarted","Data":"d9d017ba98ece5d3841f3ccf15029d183eb821767b0f0aca27a2eabe4ed859f7"}
Jul 29 20:51:44.099177 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.099144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:44.099364 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.099198    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:44.099364 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:44.099266    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:51:44.099364 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:44.099349    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:45.099331841 +0000 UTC m=+254.415820734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : configmap "kube-rbac-proxy" not found
Jul 29 20:51:44.099364 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:44.099356    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:44.099541 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:44.099395    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:45.099384098 +0000 UTC m=+254.415872990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:44.124677 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.124648    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"]
Jul 29 20:51:44.127346 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.127333    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"
Jul 29 20:51:44.130582 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.130562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-sfj2t\""
Jul 29 20:51:44.130730 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.130657    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 20:51:44.131428 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.131401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 20:51:44.157353 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.157332    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"]
Jul 29 20:51:44.200789 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.200762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z84ll\" (UniqueName: \"kubernetes.io/projected/49d8915c-5299-497a-955c-a7b9bfcd1162-kube-api-access-z84ll\") pod \"migrator-6865c9967-knsqx\" (UID: \"49d8915c-5299-497a-955c-a7b9bfcd1162\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"
Jul 29 20:51:44.301411 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.301336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z84ll\" (UniqueName: \"kubernetes.io/projected/49d8915c-5299-497a-955c-a7b9bfcd1162-kube-api-access-z84ll\") pod \"migrator-6865c9967-knsqx\" (UID: \"49d8915c-5299-497a-955c-a7b9bfcd1162\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"
Jul 29 20:51:44.308774 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.308743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z84ll\" (UniqueName: \"kubernetes.io/projected/49d8915c-5299-497a-955c-a7b9bfcd1162-kube-api-access-z84ll\") pod \"migrator-6865c9967-knsqx\" (UID: \"49d8915c-5299-497a-955c-a7b9bfcd1162\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"
Jul 29 20:51:44.436770 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.436735    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"
Jul 29 20:51:44.560813 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.560746    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx"]
Jul 29 20:51:44.563916 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:51:44.563889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d8915c_5299_497a_955c_a7b9bfcd1162.slice/crio-35b5c241c52e222a880babdb1f3a37b88667c8cb73d81aa685b262e10c7f316b WatchSource:0}: Error finding container 35b5c241c52e222a880babdb1f3a37b88667c8cb73d81aa685b262e10c7f316b: Status 404 returned error can't find the container with id 35b5c241c52e222a880babdb1f3a37b88667c8cb73d81aa685b262e10c7f316b
Jul 29 20:51:44.967776 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:44.967745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx" event={"ID":"49d8915c-5299-497a-955c-a7b9bfcd1162","Type":"ContainerStarted","Data":"35b5c241c52e222a880babdb1f3a37b88667c8cb73d81aa685b262e10c7f316b"}
Jul 29 20:51:45.108191 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:45.108155    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:45.108390 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:45.108204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:45.108390 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:45.108271    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:51:45.108390 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:45.108329    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:45.108390 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:45.108353    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:47.108337062 +0000 UTC m=+256.424825954 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : configmap "kube-rbac-proxy" not found
Jul 29 20:51:45.108390 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:45.108370    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:47.108359297 +0000 UTC m=+256.424848191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:45.380562 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:45.380539    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m4kxx_33fdf5c4-3b9d-4f59-92e4-b66c57f165ac/dns-node-resolver/0.log"
Jul 29 20:51:45.972172 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:45.972141    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx" event={"ID":"49d8915c-5299-497a-955c-a7b9bfcd1162","Type":"ContainerStarted","Data":"af2cc4a6d8968dea460e4a620f2136ce0cc4c48cb788b5c63d4804cb265ae4d6"}
Jul 29 20:51:45.972534 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:45.972201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx" event={"ID":"49d8915c-5299-497a-955c-a7b9bfcd1162","Type":"ContainerStarted","Data":"f5bdab366e6ba80b524c71e8e5c1ee0d8ed993305146e4648b89961ecd985b35"}
Jul 29 20:51:45.986234 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:45.986196    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-knsqx" podStartSLOduration=0.708194317 podStartE2EDuration="1.986181795s" totalImagesPullingTime="1.277987478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:45.985185746 +0000 UTC m=+255.301674661" watchObservedRunningTime="2026-07-29 20:51:45.986181795 +0000 UTC m=+255.302670710"
Jul 29 20:51:46.380554 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:46.380530    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9brg_a54988dd-54c1-44d6-ac26-8e9964f9794a/node-ca/0.log"
Jul 29 20:51:47.126260 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:47.126227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:47.126680 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:47.126278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:47.126680 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:47.126370    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:51:47.126680 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:47.126424    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:47.126680 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:47.126433    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:51.126418429 +0000 UTC m=+260.442907322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : configmap "kube-rbac-proxy" not found
Jul 29 20:51:47.126680 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:47.126462    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:51.126451464 +0000 UTC m=+260.442940357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:47.328331 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:47.328275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:47.328519 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:47.328417    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:47.328519 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:47.328479    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls podName:dd421a79-7fc7-46dc-9e78-3ac699f74657 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:55.328464473 +0000 UTC m=+264.644953365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pj425" (UID: "dd421a79-7fc7-46dc-9e78-3ac699f74657") : secret "cluster-monitoring-operator-tls" not found
Jul 29 20:51:47.379245 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:47.379182    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xk52g_b691b5e0-c6cc-479d-a23c-e9e96fc40efb/serve-healthcheck-canary/0.log"
Jul 29 20:51:51.158942 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:51.158900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:51.159367 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:51.158956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:51.159367 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:51.159012    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 20:51:51.159367 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:51.159048    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:51.159367 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:51.159073    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:59.159058683 +0000 UTC m=+268.475547575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : configmap "kube-rbac-proxy" not found
Jul 29 20:51:51.159367 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:51:51.159089    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls podName:dd6703c2-da11-4da2-b67c-50513265ba18 nodeName:}" failed. No retries permitted until 2026-07-29 20:51:59.159078617 +0000 UTC m=+268.475567509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7pfpq" (UID: "dd6703c2-da11-4da2-b67c-50513265ba18") : secret "insights-runtime-extractor-tls" not found
Jul 29 20:51:55.395686 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:55.395604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:55.397939 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:55.397915    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/dd421a79-7fc7-46dc-9e78-3ac699f74657-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pj425\" (UID: \"dd421a79-7fc7-46dc-9e78-3ac699f74657\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:55.556259 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:55.556217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"
Jul 29 20:51:55.676962 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:55.676894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425"]
Jul 29 20:51:55.680204 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:51:55.680182    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd421a79_7fc7_46dc_9e78_3ac699f74657.slice/crio-05004b478fd7efca27e55cd92f6736d5dbb075267a4fb72cfa55ba19e31a52b9 WatchSource:0}: Error finding container 05004b478fd7efca27e55cd92f6736d5dbb075267a4fb72cfa55ba19e31a52b9: Status 404 returned error can't find the container with id 05004b478fd7efca27e55cd92f6736d5dbb075267a4fb72cfa55ba19e31a52b9
Jul 29 20:51:56.002872 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:56.002791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425" event={"ID":"dd421a79-7fc7-46dc-9e78-3ac699f74657","Type":"ContainerStarted","Data":"05004b478fd7efca27e55cd92f6736d5dbb075267a4fb72cfa55ba19e31a52b9"}
Jul 29 20:51:58.013869 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:58.013835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425" event={"ID":"dd421a79-7fc7-46dc-9e78-3ac699f74657","Type":"ContainerStarted","Data":"aaa74a9de6851a69e4b22ab6666a09fc37e5aae408033c17a6f3da09475013f9"}
Jul 29 20:51:58.031784 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:58.031739    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pj425" podStartSLOduration=17.223561836000002 podStartE2EDuration="19.031725688s" totalImagesPullingTime="1.808163852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:51:58.031502733 +0000 UTC m=+267.347991651" watchObservedRunningTime="2026-07-29 20:51:58.031725688 +0000 UTC m=+267.348214625"
Jul 29 20:51:59.227116 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:59.227070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:59.227563 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:59.227141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:59.227689 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:59.227668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dd6703c2-da11-4da2-b67c-50513265ba18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:59.229424 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:59.229403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dd6703c2-da11-4da2-b67c-50513265ba18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7pfpq\" (UID: \"dd6703c2-da11-4da2-b67c-50513265ba18\") " pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:59.306709 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:59.306682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7pfpq"
Jul 29 20:51:59.427109 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:51:59.427081    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7pfpq"]
Jul 29 20:51:59.429849 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:51:59.429826    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd6703c2_da11_4da2_b67c_50513265ba18.slice/crio-c2aeb78c855abcbb99d39d22c7ee7d44c256dd75ce4539d4322649fd63fa6c17 WatchSource:0}: Error finding container c2aeb78c855abcbb99d39d22c7ee7d44c256dd75ce4539d4322649fd63fa6c17: Status 404 returned error can't find the container with id c2aeb78c855abcbb99d39d22c7ee7d44c256dd75ce4539d4322649fd63fa6c17
Jul 29 20:52:00.022639 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:00.022557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pfpq" event={"ID":"dd6703c2-da11-4da2-b67c-50513265ba18","Type":"ContainerStarted","Data":"c2aeb78c855abcbb99d39d22c7ee7d44c256dd75ce4539d4322649fd63fa6c17"}
Jul 29 20:52:05.043492 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:05.043449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pfpq" event={"ID":"dd6703c2-da11-4da2-b67c-50513265ba18","Type":"ContainerStarted","Data":"25ca132b8650cdccbc8ec6482eb96f2174b548abe5ead223126dc05c3dfad017"}
Jul 29 20:52:08.056114 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.056033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pfpq" event={"ID":"dd6703c2-da11-4da2-b67c-50513265ba18","Type":"ContainerStarted","Data":"e495aca4afff7197530569eb1bf6747c25cd6abd1c5df82487e88a0e1129439c"}
Jul 29 20:52:08.282654 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.282622    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"]
Jul 29 20:52:08.285586 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.285566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"
Jul 29 20:52:08.289088 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.288821    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 29 20:52:08.289088 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.288873    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wvjz8\""
Jul 29 20:52:08.290484 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.290455    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-65qpc"]
Jul 29 20:52:08.293690 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.293656    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-65qpc"
Jul 29 20:52:08.296086 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.296067    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-s5l99\""
Jul 29 20:52:08.296340 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.296209    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 20:52:08.296547 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.296526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 20:52:08.297138 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.297085    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"]
Jul 29 20:52:08.310735 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.310662    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-65qpc"]
Jul 29 20:52:08.402572 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.402525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnd6p\" (UniqueName: \"kubernetes.io/projected/4be3e202-01fd-4674-b9cc-e8f095d16563-kube-api-access-cnd6p\") pod \"downloads-7c97dc4555-65qpc\" (UID: \"4be3e202-01fd-4674-b9cc-e8f095d16563\") " pod="openshift-console/downloads-7c97dc4555-65qpc"
Jul 29 20:52:08.402758 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.402663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/764df4a6-3daf-4708-a92a-49c78f4191f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wkswh\" (UID: \"764df4a6-3daf-4708-a92a-49c78f4191f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"
Jul 29 20:52:08.503991 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.503947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/764df4a6-3daf-4708-a92a-49c78f4191f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wkswh\" (UID: \"764df4a6-3daf-4708-a92a-49c78f4191f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"
Jul 29 20:52:08.504198 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.504052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnd6p\" (UniqueName: \"kubernetes.io/projected/4be3e202-01fd-4674-b9cc-e8f095d16563-kube-api-access-cnd6p\") pod \"downloads-7c97dc4555-65qpc\" (UID: \"4be3e202-01fd-4674-b9cc-e8f095d16563\") " pod="openshift-console/downloads-7c97dc4555-65qpc"
Jul 29 20:52:08.506677 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.506648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/764df4a6-3daf-4708-a92a-49c78f4191f3-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-wkswh\" (UID: \"764df4a6-3daf-4708-a92a-49c78f4191f3\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"
Jul 29 20:52:08.513645 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.513617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnd6p\" (UniqueName: \"kubernetes.io/projected/4be3e202-01fd-4674-b9cc-e8f095d16563-kube-api-access-cnd6p\") pod \"downloads-7c97dc4555-65qpc\" (UID: \"4be3e202-01fd-4674-b9cc-e8f095d16563\") " pod="openshift-console/downloads-7c97dc4555-65qpc"
Jul 29 20:52:08.596639 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.596605    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"
Jul 29 20:52:08.606884 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.606860    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-65qpc"
Jul 29 20:52:08.748709 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.748681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"]
Jul 29 20:52:08.752535 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:08.752508    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764df4a6_3daf_4708_a92a_49c78f4191f3.slice/crio-1ddd8e5c70492acd37d06427d8ef82c8ba78c17eb37e7ad6c1c6bfd3dfdf0d75 WatchSource:0}: Error finding container 1ddd8e5c70492acd37d06427d8ef82c8ba78c17eb37e7ad6c1c6bfd3dfdf0d75: Status 404 returned error can't find the container with id 1ddd8e5c70492acd37d06427d8ef82c8ba78c17eb37e7ad6c1c6bfd3dfdf0d75
Jul 29 20:52:08.772435 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:08.772409    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-65qpc"]
Jul 29 20:52:08.777372 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:08.777334    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be3e202_01fd_4674_b9cc_e8f095d16563.slice/crio-332b8a96cb0757a4efd5910a192372c71e609324b6f9794e787faf0bc273790a WatchSource:0}: Error finding container 332b8a96cb0757a4efd5910a192372c71e609324b6f9794e787faf0bc273790a: Status 404 returned error can't find the container with id 332b8a96cb0757a4efd5910a192372c71e609324b6f9794e787faf0bc273790a
Jul 29 20:52:09.061352 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:09.061238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh" event={"ID":"764df4a6-3daf-4708-a92a-49c78f4191f3","Type":"ContainerStarted","Data":"1ddd8e5c70492acd37d06427d8ef82c8ba78c17eb37e7ad6c1c6bfd3dfdf0d75"}
Jul 29 20:52:09.062659 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:09.062623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-65qpc" event={"ID":"4be3e202-01fd-4674-b9cc-e8f095d16563","Type":"ContainerStarted","Data":"332b8a96cb0757a4efd5910a192372c71e609324b6f9794e787faf0bc273790a"}
Jul 29 20:52:11.732663 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:11.732572    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz" podUID="c51b7261-61be-4338-a660-f6b2d192525d"
Jul 29 20:52:11.732663 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:11.732594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756"
Jul 29 20:52:11.732663 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:11.732572    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z9b6g" podUID="3b8528ab-0a79-4d17-807a-597976ead2f6"
Jul 29 20:52:12.075599 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.075500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh" event={"ID":"764df4a6-3daf-4708-a92a-49c78f4191f3","Type":"ContainerStarted","Data":"c33756a3adbb4797f740aff8918356223069ae0840e2bf58036576ea1a3f23eb"}
Jul 29 20:52:12.075599 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.075567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"
Jul 29 20:52:12.078111 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.078086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:52:12.078257 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.078089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:52:12.078257 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.078082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7pfpq" event={"ID":"dd6703c2-da11-4da2-b67c-50513265ba18","Type":"ContainerStarted","Data":"6629f68e020ccfdf2195629f019a90aa3ce4b81a2ea6c65fabde3dc242d5595f"}
Jul 29 20:52:12.078395 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.078270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:52:12.082324 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.082283    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh"
Jul 29 20:52:12.091198 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.090723    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-wkswh" podStartSLOduration=1.418950387 podStartE2EDuration="4.090703285s" totalImagesPullingTime="2.671752898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:12.090076348 +0000 UTC m=+281.406565266" watchObservedRunningTime="2026-07-29 20:52:12.090703285 +0000 UTC m=+281.407192213"
Jul 29 20:52:12.107399 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:12.107241    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7pfpq" podStartSLOduration=17.210277867 podStartE2EDuration="29.10722131s" totalImagesPullingTime="11.896943443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 20:51:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:12.106534562 +0000 UTC m=+281.423023478" watchObservedRunningTime="2026-07-29 20:52:12.10722131 +0000 UTC m=+281.423710226"
Jul 29 20:52:15.063918 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.063877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:52:15.064422 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.063953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:52:15.064422 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.063984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:52:15.066721 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.066693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b8528ab-0a79-4d17-807a-597976ead2f6-metrics-tls\") pod \"dns-default-z9b6g\" (UID: \"3b8528ab-0a79-4d17-807a-597976ead2f6\") " pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:52:15.066837 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.066738    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c51b7261-61be-4338-a660-f6b2d192525d-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-l8cbz\" (UID: \"c51b7261-61be-4338-a660-f6b2d192525d\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:52:15.066837 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.066754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"image-registry-575f7f84fb-2t52z\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") " pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:52:15.081501 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.081471    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-245mr\""
Jul 29 20:52:15.081621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.081518    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgjxr\""
Jul 29 20:52:15.081621 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.081482    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ws2kf\""
Jul 29 20:52:15.088660 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.088642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:52:15.088788 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.088730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:52:15.088788 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.088762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"
Jul 29 20:52:15.290735 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.290717    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz"]
Jul 29 20:52:15.293683 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:15.293646    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51b7261_61be_4338_a660_f6b2d192525d.slice/crio-79ddf3cb81651badbb148e39f5ee0566d97189fc948b35511a190475d3d20adc WatchSource:0}: Error finding container 79ddf3cb81651badbb148e39f5ee0566d97189fc948b35511a190475d3d20adc: Status 404 returned error can't find the container with id 79ddf3cb81651badbb148e39f5ee0566d97189fc948b35511a190475d3d20adc
Jul 29 20:52:15.518959 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.518927    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z9b6g"]
Jul 29 20:52:15.523054 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.522992    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575f7f84fb-2t52z"]
Jul 29 20:52:15.524070 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:15.524041    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8528ab_0a79_4d17_807a_597976ead2f6.slice/crio-a0494c439d9335458a7559969d4e9e3e8b275003d922fe896deb9e9ecb7ebe52 WatchSource:0}: Error finding container a0494c439d9335458a7559969d4e9e3e8b275003d922fe896deb9e9ecb7ebe52: Status 404 returned error can't find the container with id a0494c439d9335458a7559969d4e9e3e8b275003d922fe896deb9e9ecb7ebe52
Jul 29 20:52:15.527119 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:15.527089    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fef84ca_71bc_4047_851d_0b83d1c25756.slice/crio-88617f791637c0ce05f5612093f54f1b6fae0110efcd531a38ee87aabd8c9f89 WatchSource:0}: Error finding container 88617f791637c0ce05f5612093f54f1b6fae0110efcd531a38ee87aabd8c9f89: Status 404 returned error can't find the container with id 88617f791637c0ce05f5612093f54f1b6fae0110efcd531a38ee87aabd8c9f89
Jul 29 20:52:15.594501 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.594272    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ff6887667-nwspg"]
Jul 29 20:52:15.597610 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.597590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.599860 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.599839    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 20:52:15.599951 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.599932    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 20:52:15.600112 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.600072    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 20:52:15.600194 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.600121    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 20:52:15.600370 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.600348    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fbn88\""
Jul 29 20:52:15.600370 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.600349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 20:52:15.606574 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.606550    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff6887667-nwspg"]
Jul 29 20:52:15.772717 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.772634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-service-ca\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.772717 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.772675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-oauth-config\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.772909 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.772732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlmx\" (UniqueName: \"kubernetes.io/projected/41830cbc-2857-41f2-8808-eb863268e7b7-kube-api-access-6vlmx\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.772909 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.772757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-console-config\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.772909 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.772828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-oauth-serving-cert\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.772909 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.772862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-serving-cert\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.873549 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.873503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-service-ca\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.873754 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.873557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-oauth-config\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.873754 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.873595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlmx\" (UniqueName: \"kubernetes.io/projected/41830cbc-2857-41f2-8808-eb863268e7b7-kube-api-access-6vlmx\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.873877 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.873821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-console-config\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.873932 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.873916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-oauth-serving-cert\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.873986 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.873957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-serving-cert\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.874447 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.874403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-service-ca\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.874578 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.874532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-console-config\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.874643 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.874610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-oauth-serving-cert\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.876607 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.876584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-oauth-config\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.876710 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.876632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-serving-cert\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.881249 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.881222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlmx\" (UniqueName: \"kubernetes.io/projected/41830cbc-2857-41f2-8808-eb863268e7b7-kube-api-access-6vlmx\") pod \"console-6ff6887667-nwspg\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") " pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:15.909307 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:15.909127    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:16.087034 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.086438    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ff6887667-nwspg"]
Jul 29 20:52:16.090868 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:16.090839    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41830cbc_2857_41f2_8808_eb863268e7b7.slice/crio-ea02f186c16818916ffffa34d4039db8fb5e9c3e3eceb4f03f2b878446488836 WatchSource:0}: Error finding container ea02f186c16818916ffffa34d4039db8fb5e9c3e3eceb4f03f2b878446488836: Status 404 returned error can't find the container with id ea02f186c16818916ffffa34d4039db8fb5e9c3e3eceb4f03f2b878446488836
Jul 29 20:52:16.098315 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.098228    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff6887667-nwspg" event={"ID":"41830cbc-2857-41f2-8808-eb863268e7b7","Type":"ContainerStarted","Data":"ea02f186c16818916ffffa34d4039db8fb5e9c3e3eceb4f03f2b878446488836"}
Jul 29 20:52:16.099673 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.099614    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz" event={"ID":"c51b7261-61be-4338-a660-f6b2d192525d","Type":"ContainerStarted","Data":"79ddf3cb81651badbb148e39f5ee0566d97189fc948b35511a190475d3d20adc"}
Jul 29 20:52:16.101733 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.101455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" event={"ID":"2fef84ca-71bc-4047-851d-0b83d1c25756","Type":"ContainerStarted","Data":"b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743"}
Jul 29 20:52:16.101733 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.101496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" event={"ID":"2fef84ca-71bc-4047-851d-0b83d1c25756","Type":"ContainerStarted","Data":"88617f791637c0ce05f5612093f54f1b6fae0110efcd531a38ee87aabd8c9f89"}
Jul 29 20:52:16.101733 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.101533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:52:16.103114 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.103067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z9b6g" event={"ID":"3b8528ab-0a79-4d17-807a-597976ead2f6","Type":"ContainerStarted","Data":"a0494c439d9335458a7559969d4e9e3e8b275003d922fe896deb9e9ecb7ebe52"}
Jul 29 20:52:16.123152 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:16.123067    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" podStartSLOduration=285.123018896 podStartE2EDuration="4m45.123018896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:16.122621882 +0000 UTC m=+285.439110798" watchObservedRunningTime="2026-07-29 20:52:16.123018896 +0000 UTC m=+285.439507810"
Jul 29 20:52:17.117219 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.117176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz" event={"ID":"c51b7261-61be-4338-a660-f6b2d192525d","Type":"ContainerStarted","Data":"d07df600a596ae781d8b876ea8cc6c5a245a10a7209f476a593c91dfe047c0e4"}
Jul 29 20:52:17.141581 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.141137    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-l8cbz" podStartSLOduration=279.767264283 podStartE2EDuration="4m41.141116589s" totalImagesPullingTime="1.373852306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:17.139868735 +0000 UTC m=+286.456357674" watchObservedRunningTime="2026-07-29 20:52:17.141116589 +0000 UTC m=+286.457605499"
Jul 29 20:52:17.416682 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.416587    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sknft"]
Jul 29 20:52:17.420280 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.420254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.422919 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.422837    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 20:52:17.422919 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.422876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 20:52:17.423282 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.423256    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 20:52:17.423562 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.423534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 20:52:17.423671 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.423585    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5q2kp\""
Jul 29 20:52:17.501276 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-root\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501276 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-wtmp\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501528 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501528 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-sys\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501528 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdxbh\" (UniqueName: \"kubernetes.io/projected/474e50f6-bd18-434d-bc8f-256acc21866b-kube-api-access-rdxbh\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501716 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-accelerators-collector-config\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501716 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-tls\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501837 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474e50f6-bd18-434d-bc8f-256acc21866b-metrics-client-ca\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.501891 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.501844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-textfile\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.602573 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.602192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-root\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.602573 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.602245    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-wtmp\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.602573 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.602269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.602573 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.602318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-sys\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.602573 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.602343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdxbh\" (UniqueName: \"kubernetes.io/projected/474e50f6-bd18-434d-bc8f-256acc21866b-kube-api-access-rdxbh\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.602573 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.602404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-accelerators-collector-config\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.602573 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.602438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-tls\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.603246 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.603105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474e50f6-bd18-434d-bc8f-256acc21866b-metrics-client-ca\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.603246 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.603162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-textfile\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.604120 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.603453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-sys\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.604120 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.603515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-root\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.604120 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.603567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-textfile\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.604120 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.603622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-wtmp\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.604120 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:17.604115    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 29 20:52:17.604501 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:17.604172    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-tls podName:474e50f6-bd18-434d-bc8f-256acc21866b nodeName:}" failed. No retries permitted until 2026-07-29 20:52:18.104154183 +0000 UTC m=+287.420643077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-tls") pod "node-exporter-sknft" (UID: "474e50f6-bd18-434d-bc8f-256acc21866b") : secret "node-exporter-tls" not found
Jul 29 20:52:17.604501 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.604411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-accelerators-collector-config\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.604870 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.604829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/474e50f6-bd18-434d-bc8f-256acc21866b-metrics-client-ca\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.607004 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.606926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:17.624862 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:17.621710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdxbh\" (UniqueName: \"kubernetes.io/projected/474e50f6-bd18-434d-bc8f-256acc21866b-kube-api-access-rdxbh\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:18.106586 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:18.106542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-tls\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:18.109680 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:18.109650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/474e50f6-bd18-434d-bc8f-256acc21866b-node-exporter-tls\") pod \"node-exporter-sknft\" (UID: \"474e50f6-bd18-434d-bc8f-256acc21866b\") " pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:18.126409 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:18.126327    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z9b6g" event={"ID":"3b8528ab-0a79-4d17-807a-597976ead2f6","Type":"ContainerStarted","Data":"94cbe05b56487044156cbeeb952acb0eab587a2b3256b4659e9642386d94366a"}
Jul 29 20:52:18.126409 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:18.126392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z9b6g" event={"ID":"3b8528ab-0a79-4d17-807a-597976ead2f6","Type":"ContainerStarted","Data":"975b7eb41e119d8aff439d7555cdf3677874a42ac14ec48dd4d513f58ccd2705"}
Jul 29 20:52:18.144380 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:18.144265    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z9b6g" podStartSLOduration=250.84004752 podStartE2EDuration="4m13.144243077s" totalImagesPullingTime="2.304195557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:18.142715786 +0000 UTC m=+287.459204704" watchObservedRunningTime="2026-07-29 20:52:18.144243077 +0000 UTC m=+287.460731993"
Jul 29 20:52:18.332884 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:18.332821    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sknft"
Jul 29 20:52:18.344614 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:18.344584    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474e50f6_bd18_434d_bc8f_256acc21866b.slice/crio-d2f9f2f569e4e2c79bbf73856e14f8c9f404003337f1468e57071acd79639f23 WatchSource:0}: Error finding container d2f9f2f569e4e2c79bbf73856e14f8c9f404003337f1468e57071acd79639f23: Status 404 returned error can't find the container with id d2f9f2f569e4e2c79bbf73856e14f8c9f404003337f1468e57071acd79639f23
Jul 29 20:52:19.132630 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.132585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sknft" event={"ID":"474e50f6-bd18-434d-bc8f-256acc21866b","Type":"ContainerStarted","Data":"d2f9f2f569e4e2c79bbf73856e14f8c9f404003337f1468e57071acd79639f23"}
Jul 29 20:52:19.133107 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.132670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:52:19.364898 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.364866    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-69cbc65479-m6t74"]
Jul 29 20:52:19.373337 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.373311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.375776 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.375745    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 29 20:52:19.375915 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.375783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2xtwv\""
Jul 29 20:52:19.375915 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.375908    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 29 20:52:19.376024 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.376004    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 29 20:52:19.376226 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.376113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cr32p71o7k5df\""
Jul 29 20:52:19.376226 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.376148    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 29 20:52:19.376463 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.376440    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 29 20:52:19.379073 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.379052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69cbc65479-m6t74"]
Jul 29 20:52:19.419491 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.419673 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.419673 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.419673 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-tls\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.419982 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5afca716-ca45-4f76-948b-9c13dcceedd1-metrics-client-ca\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.419982 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-grpc-tls\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.419982 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqfkq\" (UniqueName: \"kubernetes.io/projected/5afca716-ca45-4f76-948b-9c13dcceedd1-kube-api-access-kqfkq\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.419982 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.419823    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521211 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521258    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-tls\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5afca716-ca45-4f76-948b-9c13dcceedd1-metrics-client-ca\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-grpc-tls\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqfkq\" (UniqueName: \"kubernetes.io/projected/5afca716-ca45-4f76-948b-9c13dcceedd1-kube-api-access-kqfkq\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521662 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.521662 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.521446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.522385 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.522019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5afca716-ca45-4f76-948b-9c13dcceedd1-metrics-client-ca\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.524549 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.524515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.524549 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.524519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-tls\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.525012 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.524971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.525098 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.525034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.525323 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.525278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-grpc-tls\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.525685 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.525666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5afca716-ca45-4f76-948b-9c13dcceedd1-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.535869 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.535821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqfkq\" (UniqueName: \"kubernetes.io/projected/5afca716-ca45-4f76-948b-9c13dcceedd1-kube-api-access-kqfkq\") pod \"thanos-querier-69cbc65479-m6t74\" (UID: \"5afca716-ca45-4f76-948b-9c13dcceedd1\") " pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.686786 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.686682    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:19.876252 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:19.875760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-69cbc65479-m6t74"]
Jul 29 20:52:19.881307 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:19.877584    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afca716_ca45_4f76_948b_9c13dcceedd1.slice/crio-4b7f7aee535992a92bc94130081a6d20c95d45c0b0c85b14f4106a76628cca05 WatchSource:0}: Error finding container 4b7f7aee535992a92bc94130081a6d20c95d45c0b0c85b14f4106a76628cca05: Status 404 returned error can't find the container with id 4b7f7aee535992a92bc94130081a6d20c95d45c0b0c85b14f4106a76628cca05
Jul 29 20:52:20.138721 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:20.138684    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff6887667-nwspg" event={"ID":"41830cbc-2857-41f2-8808-eb863268e7b7","Type":"ContainerStarted","Data":"d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a"}
Jul 29 20:52:20.140185 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:20.140156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" event={"ID":"5afca716-ca45-4f76-948b-9c13dcceedd1","Type":"ContainerStarted","Data":"4b7f7aee535992a92bc94130081a6d20c95d45c0b0c85b14f4106a76628cca05"}
Jul 29 20:52:20.155244 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:20.155190    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6ff6887667-nwspg" podStartSLOduration=1.461484598 podStartE2EDuration="5.155169096s" totalImagesPullingTime="3.693684498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:20.15460869 +0000 UTC m=+289.471097596" watchObservedRunningTime="2026-07-29 20:52:20.155169096 +0000 UTC m=+289.471658012"
Jul 29 20:52:21.147320 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:21.147253    2579 generic.go:356] "Generic (PLEG): container finished" podID="474e50f6-bd18-434d-bc8f-256acc21866b" containerID="e3f6a03e62c6fa4cf724b91476e851c4c288843b2ba31f963257a0d48faa1b51" exitCode=0
Jul 29 20:52:21.147807 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:21.147543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sknft" event={"ID":"474e50f6-bd18-434d-bc8f-256acc21866b","Type":"ContainerDied","Data":"e3f6a03e62c6fa4cf724b91476e851c4c288843b2ba31f963257a0d48faa1b51"}
Jul 29 20:52:22.154672 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.154645    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sknft" event={"ID":"474e50f6-bd18-434d-bc8f-256acc21866b","Type":"ContainerStarted","Data":"25bf7c724d3cc40ce9c8c1057b5af7641a704b31f06e964c660fabd9f7ba7a8f"}
Jul 29 20:52:22.155018 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.154704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sknft" event={"ID":"474e50f6-bd18-434d-bc8f-256acc21866b","Type":"ContainerStarted","Data":"8e1bcc517de0e4ffe6f10c8a9df36062148e92c9329aae968877de79be118a98"}
Jul 29 20:52:22.156456 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.156424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" event={"ID":"5afca716-ca45-4f76-948b-9c13dcceedd1","Type":"ContainerStarted","Data":"1fc45f5905aad388ac93e1c306903b075e65891d209dac4cbdcc100c5d370968"}
Jul 29 20:52:22.165937 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.165915    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"]
Jul 29 20:52:22.168626 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.168606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:22.170863 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.170843    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 20:52:22.170988 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.170971    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v6bjv\""
Jul 29 20:52:22.178646 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.178594    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sknft" podStartSLOduration=3.067715246 podStartE2EDuration="5.178575517s" totalImagesPullingTime="2.049547134s" totalInitContainerRuntime="61.313137ms" isStatefulPod=true podCreationTimestamp="2026-07-29 20:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:22.176702403 +0000 UTC m=+291.493191329" watchObservedRunningTime="2026-07-29 20:52:22.178575517 +0000 UTC m=+291.495064433"
Jul 29 20:52:22.184152 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.182258    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"]
Jul 29 20:52:22.247769 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.247731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khqv\" (UniqueName: \"kubernetes.io/projected/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-kube-api-access-8khqv\") pod \"monitoring-plugin-68b7fc9574-tngbv\" (UID: \"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:22.248044 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.248015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tngbv\" (UID: \"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:22.251403 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:22.251372    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5nmdf" podUID="7e0f2dcf-741a-44e4-9255-331e08203b12"
Jul 29 20:52:22.349295 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.349263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tngbv\" (UID: \"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:22.349483 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.349359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khqv\" (UniqueName: \"kubernetes.io/projected/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-kube-api-access-8khqv\") pod \"monitoring-plugin-68b7fc9574-tngbv\" (UID: \"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:22.349483 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:22.349448    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 29 20:52:22.349591 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:22.349533    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-monitoring-plugin-cert podName:b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6 nodeName:}" failed. No retries permitted until 2026-07-29 20:52:22.849511621 +0000 UTC m=+292.166000532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-tngbv" (UID: "b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6") : secret "monitoring-plugin-cert" not found
Jul 29 20:52:22.357950 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.357922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khqv\" (UniqueName: \"kubernetes.io/projected/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-kube-api-access-8khqv\") pod \"monitoring-plugin-68b7fc9574-tngbv\" (UID: \"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:22.853704 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.853668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tngbv\" (UID: \"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:22.856875 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:22.856850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-tngbv\" (UID: \"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:23.095448 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:23.095407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:23.162619 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:23.162544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" event={"ID":"5afca716-ca45-4f76-948b-9c13dcceedd1","Type":"ContainerStarted","Data":"5d5b1c05e3614d6132f1c60b0fe5996c587e02dbf8b7c0fa7a3d712cc9ca2184"}
Jul 29 20:52:23.162619 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:23.162619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" event={"ID":"5afca716-ca45-4f76-948b-9c13dcceedd1","Type":"ContainerStarted","Data":"890dfabde72c7bf8d11ee655d47d087ec6dfef1516d279b1e63be23a895d1069"}
Jul 29 20:52:25.909451 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:25.909410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:25.909893 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:25.909475    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:25.911061 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:25.911034    2579 patch_prober.go:28] interesting pod/console-6ff6887667-nwspg container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body=
Jul 29 20:52:25.911193 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:25.911085    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-6ff6887667-nwspg" podUID="41830cbc-2857-41f2-8808-eb863268e7b7" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused"
Jul 29 20:52:29.142467 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:29.142429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z9b6g"
Jul 29 20:52:30.640687 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:30.640655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"]
Jul 29 20:52:30.645371 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:52:30.645350    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ce8dd9_b4c6_4bc0_a8b6_3ea22c749ee6.slice/crio-74b597eeb05551fb5acf51840c3e1c6fb075c529538ca563ab65185a176683d3 WatchSource:0}: Error finding container 74b597eeb05551fb5acf51840c3e1c6fb075c529538ca563ab65185a176683d3: Status 404 returned error can't find the container with id 74b597eeb05551fb5acf51840c3e1c6fb075c529538ca563ab65185a176683d3
Jul 29 20:52:30.651707 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:30.651685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff6887667-nwspg"]
Jul 29 20:52:31.123249 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.123223    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 20:52:31.197381 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.197329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-65qpc" event={"ID":"4be3e202-01fd-4674-b9cc-e8f095d16563","Type":"ContainerStarted","Data":"b41c5b020cc569bffffb10de620bb73a266a8b1ab8ac8cde5b236e26f82f0e26"}
Jul 29 20:52:31.197556 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.197431    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-65qpc"
Jul 29 20:52:31.200995 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.200966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" event={"ID":"5afca716-ca45-4f76-948b-9c13dcceedd1","Type":"ContainerStarted","Data":"f5b447f032b6a8c9a5ce9d17e5c410d990bab6125b9816594838343200aeb6de"}
Jul 29 20:52:31.201115 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.201039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" event={"ID":"5afca716-ca45-4f76-948b-9c13dcceedd1","Type":"ContainerStarted","Data":"bbf640f368ac5e6a4df0511fb832d71f77805f0e56d6e4380bf8041871aa7cf7"}
Jul 29 20:52:31.201115 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.201080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" event={"ID":"5afca716-ca45-4f76-948b-9c13dcceedd1","Type":"ContainerStarted","Data":"90453020d095a42ee7685a422ee158b9fc7d8af94d69fd17802c95e76c5b1e23"}
Jul 29 20:52:31.201220 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.201130    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:31.202210 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.202190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv" event={"ID":"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6","Type":"ContainerStarted","Data":"74b597eeb05551fb5acf51840c3e1c6fb075c529538ca563ab65185a176683d3"}
Jul 29 20:52:31.206834 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.206811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74"
Jul 29 20:52:31.213041 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.213017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-65qpc"
Jul 29 20:52:31.215331 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.215265    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-65qpc" podStartSLOduration=1.453515225 podStartE2EDuration="23.215247709s" totalImagesPullingTime="21.761732484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:31.214123098 +0000 UTC m=+300.530612038" watchObservedRunningTime="2026-07-29 20:52:31.215247709 +0000 UTC m=+300.531736625"
Jul 29 20:52:31.237467 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:31.237415    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-69cbc65479-m6t74" podStartSLOduration=1.823622976 podStartE2EDuration="12.237398445s" totalImagesPullingTime="10.413775469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:31.235823744 +0000 UTC m=+300.552312695" watchObservedRunningTime="2026-07-29 20:52:31.237398445 +0000 UTC m=+300.553887361"
Jul 29 20:52:33.214796 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:33.214745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv" event={"ID":"b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6","Type":"ContainerStarted","Data":"e6049aa50a1af908e18fa3ba95774c490d2c4b975922d7512b3db4e2efa3f084"}
Jul 29 20:52:33.215373 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:33.215339    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:33.220042 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:33.220020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv"
Jul 29 20:52:33.230462 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:33.230411    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-tngbv" podStartSLOduration=8.834367521 podStartE2EDuration="11.230376576s" totalImagesPullingTime="2.396009055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:52:33.229194473 +0000 UTC m=+302.545683389" watchObservedRunningTime="2026-07-29 20:52:33.230376576 +0000 UTC m=+302.546865491"
Jul 29 20:52:34.250548 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:34.250482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:52:35.093880 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:35.093838    2579 patch_prober.go:28] interesting pod/image-registry-575f7f84fb-2t52z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 20:52:35.094053 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:35.093901    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 20:52:37.122887 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:37.122853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:52:49.275968 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:49.275933    2579 generic.go:356] "Generic (PLEG): container finished" podID="c1034613-bed5-4ad8-908c-7aa2cdde9e1c" containerID="7faceed7dff5fc24c10800f642ad6b4502d62bf3859e1882c4e3096cb6f59f6f" exitCode=0
Jul 29 20:52:49.276508 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:49.276006    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2" event={"ID":"c1034613-bed5-4ad8-908c-7aa2cdde9e1c","Type":"ContainerDied","Data":"7faceed7dff5fc24c10800f642ad6b4502d62bf3859e1882c4e3096cb6f59f6f"}
Jul 29 20:52:49.276508 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:49.276372    2579 scope.go:117] "RemoveContainer" containerID="7faceed7dff5fc24c10800f642ad6b4502d62bf3859e1882c4e3096cb6f59f6f"
Jul 29 20:52:50.280839 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:50.280808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2" event={"ID":"c1034613-bed5-4ad8-908c-7aa2cdde9e1c","Type":"ContainerStarted","Data":"36f8e729624312238ffbbbbdc6b593e22a7650039bd070e503eacdfb3a44acc0"}
Jul 29 20:52:53.292563 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:53.292476    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ffc6d67-25c5-4c0b-9d7e-693c2a790be1" containerID="cf23e31d3de995968d9acd1ec061a64064555fd01126b25594b498ac8d3de920" exitCode=0
Jul 29 20:52:53.292563 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:53.292521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-m7wjk" event={"ID":"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1","Type":"ContainerDied","Data":"cf23e31d3de995968d9acd1ec061a64064555fd01126b25594b498ac8d3de920"}
Jul 29 20:52:53.293020 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:53.292818    2579 scope.go:117] "RemoveContainer" containerID="cf23e31d3de995968d9acd1ec061a64064555fd01126b25594b498ac8d3de920"
Jul 29 20:52:54.298863 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:54.298821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-m7wjk" event={"ID":"4ffc6d67-25c5-4c0b-9d7e-693c2a790be1","Type":"ContainerStarted","Data":"31166c54737844fd18d45bd97392e7c8a75f22bf9719e12da89432f7416314a8"}
Jul 29 20:52:55.682997 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:55.682948    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6ff6887667-nwspg" podUID="41830cbc-2857-41f2-8808-eb863268e7b7" containerName="console" containerID="cri-o://d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a" gracePeriod=15
Jul 29 20:52:55.958231 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:55.958165    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:56.143944 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.143917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-oauth-serving-cert\") pod \"41830cbc-2857-41f2-8808-eb863268e7b7\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") "
Jul 29 20:52:56.144106 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.143952    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-service-ca\") pod \"41830cbc-2857-41f2-8808-eb863268e7b7\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") "
Jul 29 20:52:56.144106 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.143976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-console-config\") pod \"41830cbc-2857-41f2-8808-eb863268e7b7\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") "
Jul 29 20:52:56.144106 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-serving-cert\") pod \"41830cbc-2857-41f2-8808-eb863268e7b7\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") "
Jul 29 20:52:56.144248 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-oauth-config\") pod \"41830cbc-2857-41f2-8808-eb863268e7b7\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") "
Jul 29 20:52:56.144248 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vlmx\" (UniqueName: \"kubernetes.io/projected/41830cbc-2857-41f2-8808-eb863268e7b7-kube-api-access-6vlmx\") pod \"41830cbc-2857-41f2-8808-eb863268e7b7\" (UID: \"41830cbc-2857-41f2-8808-eb863268e7b7\") "
Jul 29 20:52:56.144390 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "41830cbc-2857-41f2-8808-eb863268e7b7" (UID: "41830cbc-2857-41f2-8808-eb863268e7b7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:52:56.144449 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144384    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-console-config" (OuterVolumeSpecName: "console-config") pod "41830cbc-2857-41f2-8808-eb863268e7b7" (UID: "41830cbc-2857-41f2-8808-eb863268e7b7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:52:56.144449 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144403    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-service-ca" (OuterVolumeSpecName: "service-ca") pod "41830cbc-2857-41f2-8808-eb863268e7b7" (UID: "41830cbc-2857-41f2-8808-eb863268e7b7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:52:56.144534 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144484    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-oauth-serving-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:52:56.144534 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144503    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:52:56.144534 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.144517    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/41830cbc-2857-41f2-8808-eb863268e7b7-console-config\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:52:56.146254 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.146223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41830cbc-2857-41f2-8808-eb863268e7b7-kube-api-access-6vlmx" (OuterVolumeSpecName: "kube-api-access-6vlmx") pod "41830cbc-2857-41f2-8808-eb863268e7b7" (UID: "41830cbc-2857-41f2-8808-eb863268e7b7"). InnerVolumeSpecName "kube-api-access-6vlmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:52:56.146461 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.146275    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "41830cbc-2857-41f2-8808-eb863268e7b7" (UID: "41830cbc-2857-41f2-8808-eb863268e7b7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:52:56.146461 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.146429    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "41830cbc-2857-41f2-8808-eb863268e7b7" (UID: "41830cbc-2857-41f2-8808-eb863268e7b7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:52:56.245261 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.245187    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-serving-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:52:56.245261 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.245209    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/41830cbc-2857-41f2-8808-eb863268e7b7-console-oauth-config\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:52:56.245261 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.245218    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vlmx\" (UniqueName: \"kubernetes.io/projected/41830cbc-2857-41f2-8808-eb863268e7b7-kube-api-access-6vlmx\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:52:56.311779 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.311750    2579 generic.go:356] "Generic (PLEG): container finished" podID="41830cbc-2857-41f2-8808-eb863268e7b7" containerID="d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a" exitCode=0
Jul 29 20:52:56.311918 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.311787    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ff6887667-nwspg"
Jul 29 20:52:56.311918 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.311834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff6887667-nwspg" event={"ID":"41830cbc-2857-41f2-8808-eb863268e7b7","Type":"ContainerDied","Data":"d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a"}
Jul 29 20:52:56.311918 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.311874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ff6887667-nwspg" event={"ID":"41830cbc-2857-41f2-8808-eb863268e7b7","Type":"ContainerDied","Data":"ea02f186c16818916ffffa34d4039db8fb5e9c3e3eceb4f03f2b878446488836"}
Jul 29 20:52:56.311918 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.311895    2579 scope.go:117] "RemoveContainer" containerID="d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a"
Jul 29 20:52:56.320714 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.320696    2579 scope.go:117] "RemoveContainer" containerID="d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a"
Jul 29 20:52:56.320976 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:52:56.320956    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a\": container with ID starting with d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a not found: ID does not exist" containerID="d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a"
Jul 29 20:52:56.321043 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.320988    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a"} err="failed to get container status \"d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a\": rpc error: code = NotFound desc = could not find container \"d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a\": container with ID starting with d67c7f3823d4787090664deaa77d3e49fc49977f608e790d8bbbf5b070aedc4a not found: ID does not exist"
Jul 29 20:52:56.331706 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.331677    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ff6887667-nwspg"]
Jul 29 20:52:56.336918 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:56.336899    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ff6887667-nwspg"]
Jul 29 20:52:57.254613 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:52:57.254580    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41830cbc-2857-41f2-8808-eb863268e7b7" path="/var/lib/kubelet/pods/41830cbc-2857-41f2-8808-eb863268e7b7/volumes"
Jul 29 20:53:08.757124 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:08.757091    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575f7f84fb-2t52z"]
Jul 29 20:53:33.775762 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:33.775698    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756" containerName="registry" containerID="cri-o://b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743" gracePeriod=30
Jul 29 20:53:35.021111 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.021086    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:53:35.141072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.140980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-installation-pull-secrets\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141013    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fef84ca-71bc-4047-851d-0b83d1c25756-ca-trust-extracted\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141038    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-trusted-ca\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141072 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141061    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwpfr\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-kube-api-access-pwpfr\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141421 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141421 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141121    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-image-registry-private-configuration\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141421 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141162    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-certificates\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141421 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141205    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-bound-sa-token\") pod \"2fef84ca-71bc-4047-851d-0b83d1c25756\" (UID: \"2fef84ca-71bc-4047-851d-0b83d1c25756\") "
Jul 29 20:53:35.141628 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141571    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:53:35.142028 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.141999    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:53:35.143578 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.143544    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:53:35.143684 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.143607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:53:35.143852 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.143834    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:53:35.144042 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.144014    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:53:35.144128 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.144083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-kube-api-access-pwpfr" (OuterVolumeSpecName: "kube-api-access-pwpfr") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "kube-api-access-pwpfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:53:35.149514 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.149489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fef84ca-71bc-4047-851d-0b83d1c25756-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2fef84ca-71bc-4047-851d-0b83d1c25756" (UID: "2fef84ca-71bc-4047-851d-0b83d1c25756"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:53:35.242170 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242137    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-installation-pull-secrets\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.242170 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242167    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2fef84ca-71bc-4047-851d-0b83d1c25756-ca-trust-extracted\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.242170 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242177    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-trusted-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.242371 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242188    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpfr\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-kube-api-access-pwpfr\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.242371 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242198    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-tls\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.242371 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242206    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2fef84ca-71bc-4047-851d-0b83d1c25756-image-registry-private-configuration\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.242371 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242215    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2fef84ca-71bc-4047-851d-0b83d1c25756-registry-certificates\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.242371 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.242223    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2fef84ca-71bc-4047-851d-0b83d1c25756-bound-sa-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:53:35.452452 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.452386    2579 generic.go:356] "Generic (PLEG): container finished" podID="2fef84ca-71bc-4047-851d-0b83d1c25756" containerID="b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743" exitCode=0
Jul 29 20:53:35.452452 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.452434    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z"
Jul 29 20:53:35.452592 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.452453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" event={"ID":"2fef84ca-71bc-4047-851d-0b83d1c25756","Type":"ContainerDied","Data":"b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743"}
Jul 29 20:53:35.452592 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.452497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575f7f84fb-2t52z" event={"ID":"2fef84ca-71bc-4047-851d-0b83d1c25756","Type":"ContainerDied","Data":"88617f791637c0ce05f5612093f54f1b6fae0110efcd531a38ee87aabd8c9f89"}
Jul 29 20:53:35.452592 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.452524    2579 scope.go:117] "RemoveContainer" containerID="b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743"
Jul 29 20:53:35.460507 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.460490    2579 scope.go:117] "RemoveContainer" containerID="b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743"
Jul 29 20:53:35.460746 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:53:35.460728    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743\": container with ID starting with b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743 not found: ID does not exist" containerID="b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743"
Jul 29 20:53:35.460805 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.460753    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743"} err="failed to get container status \"b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743\": rpc error: code = NotFound desc = could not find container \"b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743\": container with ID starting with b60ee78e91ab0ab39db694136a65555b1ba01155971de7ffdfca102d279b1743 not found: ID does not exist"
Jul 29 20:53:35.467905 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.467883    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575f7f84fb-2t52z"]
Jul 29 20:53:35.471663 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:35.471645    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-575f7f84fb-2t52z"]
Jul 29 20:53:37.254565 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.254537    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756" path="/var/lib/kubelet/pods/2fef84ca-71bc-4047-851d-0b83d1c25756/volumes"
Jul 29 20:53:37.552819 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.552752    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b7bfbb774-wzqgn"]
Jul 29 20:53:37.553028 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.553017    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41830cbc-2857-41f2-8808-eb863268e7b7" containerName="console"
Jul 29 20:53:37.553069 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.553030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41830cbc-2857-41f2-8808-eb863268e7b7" containerName="console"
Jul 29 20:53:37.553069 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.553048    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756" containerName="registry"
Jul 29 20:53:37.553069 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.553053    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756" containerName="registry"
Jul 29 20:53:37.553155 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.553097    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41830cbc-2857-41f2-8808-eb863268e7b7" containerName="console"
Jul 29 20:53:37.553155 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.553106    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fef84ca-71bc-4047-851d-0b83d1c25756" containerName="registry"
Jul 29 20:53:37.556550 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.556536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.558686 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.558665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 20:53:37.559396 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.559379    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 20:53:37.559585 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.559571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 20:53:37.559662 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.559614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 20:53:37.559727 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.559668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 20:53:37.559727 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.559714    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fbn88\""
Jul 29 20:53:37.569303 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.566220    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 20:53:37.569303 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.567852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b7bfbb774-wzqgn"]
Jul 29 20:53:37.659179 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.659148    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqvx\" (UniqueName: \"kubernetes.io/projected/a80c20c9-f60c-4d7c-a2a9-73e437946295-kube-api-access-mvqvx\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.659179 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.659180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-serving-cert\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.659384 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.659203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-service-ca\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.659384 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.659218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-oauth-serving-cert\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.659384 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.659265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-trusted-ca-bundle\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.659384 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.659316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-oauth-config\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.659384 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.659350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-config\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.759869 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.759843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-config\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.759997 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.759875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqvx\" (UniqueName: \"kubernetes.io/projected/a80c20c9-f60c-4d7c-a2a9-73e437946295-kube-api-access-mvqvx\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.759997 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.759893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-serving-cert\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.759997 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.759913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-service-ca\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.760148 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.760025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-oauth-serving-cert\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.760148 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.760078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-trusted-ca-bundle\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.760148 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.760118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-oauth-config\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.760670 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.760646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-config\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.760775 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.760669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-service-ca\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.760830 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.760763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-oauth-serving-cert\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.760964 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.760945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-trusted-ca-bundle\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.762379 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.762362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-oauth-config\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.762438 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.762400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-serving-cert\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.768788 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.768766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqvx\" (UniqueName: \"kubernetes.io/projected/a80c20c9-f60c-4d7c-a2a9-73e437946295-kube-api-access-mvqvx\") pod \"console-6b7bfbb774-wzqgn\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") " pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.866943 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.866920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:37.989976 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:37.989956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b7bfbb774-wzqgn"]
Jul 29 20:53:37.992233 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:53:37.992210    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80c20c9_f60c_4d7c_a2a9_73e437946295.slice/crio-8211b005288661868516d8ec7dd0fab46e2c816eeebc221ffc483e2feac03e59 WatchSource:0}: Error finding container 8211b005288661868516d8ec7dd0fab46e2c816eeebc221ffc483e2feac03e59: Status 404 returned error can't find the container with id 8211b005288661868516d8ec7dd0fab46e2c816eeebc221ffc483e2feac03e59
Jul 29 20:53:38.465355 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:38.465325    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7bfbb774-wzqgn" event={"ID":"a80c20c9-f60c-4d7c-a2a9-73e437946295","Type":"ContainerStarted","Data":"7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48"}
Jul 29 20:53:38.465727 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:38.465374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7bfbb774-wzqgn" event={"ID":"a80c20c9-f60c-4d7c-a2a9-73e437946295","Type":"ContainerStarted","Data":"8211b005288661868516d8ec7dd0fab46e2c816eeebc221ffc483e2feac03e59"}
Jul 29 20:53:38.483339 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:38.483283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b7bfbb774-wzqgn" podStartSLOduration=1.483268031 podStartE2EDuration="1.483268031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:38.482502151 +0000 UTC m=+367.798991066" watchObservedRunningTime="2026-07-29 20:53:38.483268031 +0000 UTC m=+367.799756946"
Jul 29 20:53:45.114945 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:45.114908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:53:45.117090 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:45.117069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e0f2dcf-741a-44e4-9255-331e08203b12-metrics-certs\") pod \"network-metrics-daemon-5nmdf\" (UID: \"7e0f2dcf-741a-44e4-9255-331e08203b12\") " pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:53:45.353501 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:45.353474    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vp87c\""
Jul 29 20:53:45.362170 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:45.362154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5nmdf"
Jul 29 20:53:45.484930 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:45.484899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5nmdf"]
Jul 29 20:53:45.487895 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:53:45.487866    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0f2dcf_741a_44e4_9255_331e08203b12.slice/crio-c9324ff1b67479e7a5ad7f6442af142efbc587b4371ef2374b30a6d250584715 WatchSource:0}: Error finding container c9324ff1b67479e7a5ad7f6442af142efbc587b4371ef2374b30a6d250584715: Status 404 returned error can't find the container with id c9324ff1b67479e7a5ad7f6442af142efbc587b4371ef2374b30a6d250584715
Jul 29 20:53:45.489798 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:45.489780    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 20:53:46.499409 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:46.499373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5nmdf" event={"ID":"7e0f2dcf-741a-44e4-9255-331e08203b12","Type":"ContainerStarted","Data":"26ce436239e0314bdf5b5757bf8bc7c3420d1cbf7b08586540c91a7f93ddb4e6"}
Jul 29 20:53:46.499793 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:46.499456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5nmdf" event={"ID":"7e0f2dcf-741a-44e4-9255-331e08203b12","Type":"ContainerStarted","Data":"fca7727d8a240074808b6258354a249c440f48f7358bad64b3762b2d7d6ece55"}
Jul 29 20:53:46.499793 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:46.499475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5nmdf" event={"ID":"7e0f2dcf-741a-44e4-9255-331e08203b12","Type":"ContainerStarted","Data":"c9324ff1b67479e7a5ad7f6442af142efbc587b4371ef2374b30a6d250584715"}
Jul 29 20:53:46.515946 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:46.515906    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5nmdf" podStartSLOduration=374.713102764 podStartE2EDuration="6m15.515892026s" totalImagesPullingTime="802.789262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:53:46.51401134 +0000 UTC m=+375.830500256" watchObservedRunningTime="2026-07-29 20:53:46.515892026 +0000 UTC m=+375.832380940"
Jul 29 20:53:47.867067 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:47.867039    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:47.867468 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:47.867090    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:47.871610 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:47.871591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:53:48.510731 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:53:48.510700    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:54:39.602308 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.602261    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"]
Jul 29 20:54:39.605575 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.605559    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.608297 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.608266    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 20:54:39.609131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.609116    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 20:54:39.609131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.609122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cpktn\""
Jul 29 20:54:39.615097 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.615078    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"]
Jul 29 20:54:39.678602 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.678575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.678692 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.678612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxbgg\" (UniqueName: \"kubernetes.io/projected/58740960-9af0-4037-a434-a2e2b7d04661-kube-api-access-rxbgg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.678692 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.678638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.779584 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.779557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.779684 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.779597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxbgg\" (UniqueName: \"kubernetes.io/projected/58740960-9af0-4037-a434-a2e2b7d04661-kube-api-access-rxbgg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.779749 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.779713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.779889 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.779871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.779964 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.779949    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.788103 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.788079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxbgg\" (UniqueName: \"kubernetes.io/projected/58740960-9af0-4037-a434-a2e2b7d04661-kube-api-access-rxbgg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:39.914520 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:39.914469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:40.041752 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:40.041718    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"]
Jul 29 20:54:40.045784 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:54:40.045757    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58740960_9af0_4037_a434_a2e2b7d04661.slice/crio-2f2bc92042e00b2797795e8c02371ca70f372060eb3e0909a0c84102293cf70e WatchSource:0}: Error finding container 2f2bc92042e00b2797795e8c02371ca70f372060eb3e0909a0c84102293cf70e: Status 404 returned error can't find the container with id 2f2bc92042e00b2797795e8c02371ca70f372060eb3e0909a0c84102293cf70e
Jul 29 20:54:40.697566 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:40.697530    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz" event={"ID":"58740960-9af0-4037-a434-a2e2b7d04661","Type":"ContainerStarted","Data":"2f2bc92042e00b2797795e8c02371ca70f372060eb3e0909a0c84102293cf70e"}
Jul 29 20:54:46.722364 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:46.722331    2579 generic.go:356] "Generic (PLEG): container finished" podID="58740960-9af0-4037-a434-a2e2b7d04661" containerID="cbf6cc75b6aa01d0ed3bd6acec229f1f1494140386027a5ae385d8a5dff12887" exitCode=0
Jul 29 20:54:46.722732 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:46.722428    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz" event={"ID":"58740960-9af0-4037-a434-a2e2b7d04661","Type":"ContainerDied","Data":"cbf6cc75b6aa01d0ed3bd6acec229f1f1494140386027a5ae385d8a5dff12887"}
Jul 29 20:54:48.732696 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:48.732666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz" event={"ID":"58740960-9af0-4037-a434-a2e2b7d04661","Type":"ContainerStarted","Data":"d0ba25d90cd9ee87b9ce321fa514d4b3c935bf8c2a684e9729a7eef0ae466019"}
Jul 29 20:54:49.737840 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:49.737808    2579 generic.go:356] "Generic (PLEG): container finished" podID="58740960-9af0-4037-a434-a2e2b7d04661" containerID="d0ba25d90cd9ee87b9ce321fa514d4b3c935bf8c2a684e9729a7eef0ae466019" exitCode=0
Jul 29 20:54:49.738271 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:49.737899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz" event={"ID":"58740960-9af0-4037-a434-a2e2b7d04661","Type":"ContainerDied","Data":"d0ba25d90cd9ee87b9ce321fa514d4b3c935bf8c2a684e9729a7eef0ae466019"}
Jul 29 20:54:56.770878 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:56.770842    2579 generic.go:356] "Generic (PLEG): container finished" podID="58740960-9af0-4037-a434-a2e2b7d04661" containerID="5af882840b4e0454a815800127b4a88cf46cead342e3c2b645be8b829d868e5e" exitCode=0
Jul 29 20:54:56.771220 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:56.770921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz" event={"ID":"58740960-9af0-4037-a434-a2e2b7d04661","Type":"ContainerDied","Data":"5af882840b4e0454a815800127b4a88cf46cead342e3c2b645be8b829d868e5e"}
Jul 29 20:54:57.899964 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:57.899940    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:58.004088 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.004065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxbgg\" (UniqueName: \"kubernetes.io/projected/58740960-9af0-4037-a434-a2e2b7d04661-kube-api-access-rxbgg\") pod \"58740960-9af0-4037-a434-a2e2b7d04661\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") "
Jul 29 20:54:58.004226 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.004110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-util\") pod \"58740960-9af0-4037-a434-a2e2b7d04661\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") "
Jul 29 20:54:58.004226 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.004127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-bundle\") pod \"58740960-9af0-4037-a434-a2e2b7d04661\" (UID: \"58740960-9af0-4037-a434-a2e2b7d04661\") "
Jul 29 20:54:58.004758 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.004736    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-bundle" (OuterVolumeSpecName: "bundle") pod "58740960-9af0-4037-a434-a2e2b7d04661" (UID: "58740960-9af0-4037-a434-a2e2b7d04661"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:54:58.006158 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.006137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58740960-9af0-4037-a434-a2e2b7d04661-kube-api-access-rxbgg" (OuterVolumeSpecName: "kube-api-access-rxbgg") pod "58740960-9af0-4037-a434-a2e2b7d04661" (UID: "58740960-9af0-4037-a434-a2e2b7d04661"). InnerVolumeSpecName "kube-api-access-rxbgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:54:58.009645 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.009612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-util" (OuterVolumeSpecName: "util") pod "58740960-9af0-4037-a434-a2e2b7d04661" (UID: "58740960-9af0-4037-a434-a2e2b7d04661"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:54:58.104837 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.104815    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxbgg\" (UniqueName: \"kubernetes.io/projected/58740960-9af0-4037-a434-a2e2b7d04661-kube-api-access-rxbgg\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:54:58.104837 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.104836    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-util\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:54:58.104970 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.104845    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58740960-9af0-4037-a434-a2e2b7d04661-bundle\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:54:58.779382 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.779356    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz"
Jul 29 20:54:58.779553 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.779353    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540n7rnz" event={"ID":"58740960-9af0-4037-a434-a2e2b7d04661","Type":"ContainerDied","Data":"2f2bc92042e00b2797795e8c02371ca70f372060eb3e0909a0c84102293cf70e"}
Jul 29 20:54:58.779553 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:54:58.779470    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f2bc92042e00b2797795e8c02371ca70f372060eb3e0909a0c84102293cf70e"
Jul 29 20:55:02.345980 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.345949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"]
Jul 29 20:55:02.346433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.346249    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58740960-9af0-4037-a434-a2e2b7d04661" containerName="pull"
Jul 29 20:55:02.346433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.346260    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58740960-9af0-4037-a434-a2e2b7d04661" containerName="pull"
Jul 29 20:55:02.346433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.346267    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58740960-9af0-4037-a434-a2e2b7d04661" containerName="extract"
Jul 29 20:55:02.346433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.346273    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58740960-9af0-4037-a434-a2e2b7d04661" containerName="extract"
Jul 29 20:55:02.346433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.346281    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58740960-9af0-4037-a434-a2e2b7d04661" containerName="util"
Jul 29 20:55:02.346433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.346301    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58740960-9af0-4037-a434-a2e2b7d04661" containerName="util"
Jul 29 20:55:02.346433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.346364    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58740960-9af0-4037-a434-a2e2b7d04661" containerName="extract"
Jul 29 20:55:02.361854 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.361833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.364770 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.364741    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2v2kx\""
Jul 29 20:55:02.364890 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.364765    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:55:02.364961 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.364942    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 29 20:55:02.365524 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.365500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"]
Jul 29 20:55:02.435231 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.435198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdqj\" (UniqueName: \"kubernetes.io/projected/5c89987a-d264-47b5-aa47-ba0721f20ab6-kube-api-access-tgdqj\") pod \"cert-manager-operator-controller-manager-55c444bf86-zpzhg\" (UID: \"5c89987a-d264-47b5-aa47-ba0721f20ab6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.435403 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.435245    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c89987a-d264-47b5-aa47-ba0721f20ab6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zpzhg\" (UID: \"5c89987a-d264-47b5-aa47-ba0721f20ab6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.536278 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.536249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdqj\" (UniqueName: \"kubernetes.io/projected/5c89987a-d264-47b5-aa47-ba0721f20ab6-kube-api-access-tgdqj\") pod \"cert-manager-operator-controller-manager-55c444bf86-zpzhg\" (UID: \"5c89987a-d264-47b5-aa47-ba0721f20ab6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.536433 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.536306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c89987a-d264-47b5-aa47-ba0721f20ab6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zpzhg\" (UID: \"5c89987a-d264-47b5-aa47-ba0721f20ab6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.536611 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.536596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c89987a-d264-47b5-aa47-ba0721f20ab6-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-zpzhg\" (UID: \"5c89987a-d264-47b5-aa47-ba0721f20ab6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.545123 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.545101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdqj\" (UniqueName: \"kubernetes.io/projected/5c89987a-d264-47b5-aa47-ba0721f20ab6-kube-api-access-tgdqj\") pod \"cert-manager-operator-controller-manager-55c444bf86-zpzhg\" (UID: \"5c89987a-d264-47b5-aa47-ba0721f20ab6\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.672830 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.672769    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"
Jul 29 20:55:02.858561 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:02.858538    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg"]
Jul 29 20:55:02.860812 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:55:02.860787    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c89987a_d264_47b5_aa47_ba0721f20ab6.slice/crio-17cb5ab2dad2df4dfea1ac94cef839b798a0dc26591a9170c24bb099cf03763d WatchSource:0}: Error finding container 17cb5ab2dad2df4dfea1ac94cef839b798a0dc26591a9170c24bb099cf03763d: Status 404 returned error can't find the container with id 17cb5ab2dad2df4dfea1ac94cef839b798a0dc26591a9170c24bb099cf03763d
Jul 29 20:55:03.798991 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:03.798954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg" event={"ID":"5c89987a-d264-47b5-aa47-ba0721f20ab6","Type":"ContainerStarted","Data":"17cb5ab2dad2df4dfea1ac94cef839b798a0dc26591a9170c24bb099cf03763d"}
Jul 29 20:55:06.813593 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:06.813552    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg" event={"ID":"5c89987a-d264-47b5-aa47-ba0721f20ab6","Type":"ContainerStarted","Data":"6835d1c71f84a944b21be045391211fc26a19447f3f3f84dbf638893e4092e26"}
Jul 29 20:55:06.833506 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:06.833461    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-zpzhg" podStartSLOduration=1.777050067 podStartE2EDuration="4.833443567s" totalImagesPullingTime="3.0563935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:06.831734446 +0000 UTC m=+456.148223361" watchObservedRunningTime="2026-07-29 20:55:06.833443567 +0000 UTC m=+456.149932484"
Jul 29 20:55:07.133272 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.133245    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-54t75"]
Jul 29 20:55:07.154730 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.154705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-54t75"]
Jul 29 20:55:07.154885 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.154785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.157005 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.156985    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 20:55:07.157005 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.156996    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 20:55:07.157252 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.157238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4g8hp\""
Jul 29 20:55:07.270644 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.270614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wtt\" (UniqueName: \"kubernetes.io/projected/d14eb7eb-bfa0-47c8-a334-6a4fb5e05591-kube-api-access-m9wtt\") pod \"cert-manager-webhook-6486ffd997-54t75\" (UID: \"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591\") " pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.270782 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.270666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d14eb7eb-bfa0-47c8-a334-6a4fb5e05591-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-54t75\" (UID: \"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591\") " pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.371407 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.371381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wtt\" (UniqueName: \"kubernetes.io/projected/d14eb7eb-bfa0-47c8-a334-6a4fb5e05591-kube-api-access-m9wtt\") pod \"cert-manager-webhook-6486ffd997-54t75\" (UID: \"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591\") " pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.371549 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.371434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d14eb7eb-bfa0-47c8-a334-6a4fb5e05591-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-54t75\" (UID: \"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591\") " pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.384405 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.384352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d14eb7eb-bfa0-47c8-a334-6a4fb5e05591-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-54t75\" (UID: \"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591\") " pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.391268 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.391242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wtt\" (UniqueName: \"kubernetes.io/projected/d14eb7eb-bfa0-47c8-a334-6a4fb5e05591-kube-api-access-m9wtt\") pod \"cert-manager-webhook-6486ffd997-54t75\" (UID: \"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591\") " pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.464123 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.464097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:07.633565 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.633539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-54t75"]
Jul 29 20:55:07.636014 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:55:07.635954    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd14eb7eb_bfa0_47c8_a334_6a4fb5e05591.slice/crio-b9b3fcb0c4eb33331a619f57eab2ad658cc4516aa6799dcdaa96f24b3e701141 WatchSource:0}: Error finding container b9b3fcb0c4eb33331a619f57eab2ad658cc4516aa6799dcdaa96f24b3e701141: Status 404 returned error can't find the container with id b9b3fcb0c4eb33331a619f57eab2ad658cc4516aa6799dcdaa96f24b3e701141
Jul 29 20:55:07.819067 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:07.819038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-54t75" event={"ID":"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591","Type":"ContainerStarted","Data":"b9b3fcb0c4eb33331a619f57eab2ad658cc4516aa6799dcdaa96f24b3e701141"}
Jul 29 20:55:09.231749 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.231715    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"]
Jul 29 20:55:09.252537 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.252510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.254681 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.254660    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7dm6z\""
Jul 29 20:55:09.259506 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.259485    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"]
Jul 29 20:55:09.290760 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.290737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxcr\" (UniqueName: \"kubernetes.io/projected/f4e626f0-7f5c-4d5b-825e-383568363f1f-kube-api-access-fcxcr\") pod \"cert-manager-cainjector-559cdd8c87-7rcxk\" (UID: \"f4e626f0-7f5c-4d5b-825e-383568363f1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.290863 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.290799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4e626f0-7f5c-4d5b-825e-383568363f1f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7rcxk\" (UID: \"f4e626f0-7f5c-4d5b-825e-383568363f1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.391932 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.391895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxcr\" (UniqueName: \"kubernetes.io/projected/f4e626f0-7f5c-4d5b-825e-383568363f1f-kube-api-access-fcxcr\") pod \"cert-manager-cainjector-559cdd8c87-7rcxk\" (UID: \"f4e626f0-7f5c-4d5b-825e-383568363f1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.392102 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.391958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4e626f0-7f5c-4d5b-825e-383568363f1f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7rcxk\" (UID: \"f4e626f0-7f5c-4d5b-825e-383568363f1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.401065 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.401041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f4e626f0-7f5c-4d5b-825e-383568363f1f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7rcxk\" (UID: \"f4e626f0-7f5c-4d5b-825e-383568363f1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.401474 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.401455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxcr\" (UniqueName: \"kubernetes.io/projected/f4e626f0-7f5c-4d5b-825e-383568363f1f-kube-api-access-fcxcr\") pod \"cert-manager-cainjector-559cdd8c87-7rcxk\" (UID: \"f4e626f0-7f5c-4d5b-825e-383568363f1f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.565095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.565016    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"
Jul 29 20:55:09.708252 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:09.708226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk"]
Jul 29 20:55:10.545255 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:55:10.545225    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4e626f0_7f5c_4d5b_825e_383568363f1f.slice/crio-45f66a981df4a710f5d7d616b8835c326ac997991b081d7ff702dbf2b37adcb7 WatchSource:0}: Error finding container 45f66a981df4a710f5d7d616b8835c326ac997991b081d7ff702dbf2b37adcb7: Status 404 returned error can't find the container with id 45f66a981df4a710f5d7d616b8835c326ac997991b081d7ff702dbf2b37adcb7
Jul 29 20:55:10.834861 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:10.834782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-54t75" event={"ID":"d14eb7eb-bfa0-47c8-a334-6a4fb5e05591","Type":"ContainerStarted","Data":"fc33b8ca11ffad788accea219438ce37b4b80802a7ae43a4d511dd6093cc6ec9"}
Jul 29 20:55:10.834861 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:10.834843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:10.840807 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:10.840782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk" event={"ID":"f4e626f0-7f5c-4d5b-825e-383568363f1f","Type":"ContainerStarted","Data":"45f66a981df4a710f5d7d616b8835c326ac997991b081d7ff702dbf2b37adcb7"}
Jul 29 20:55:10.852602 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:10.852559    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-54t75" podStartSLOduration=0.906866938 podStartE2EDuration="3.85254835s" totalImagesPullingTime="2.945681412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:10.851978556 +0000 UTC m=+460.168467471" watchObservedRunningTime="2026-07-29 20:55:10.85254835 +0000 UTC m=+460.169037266"
Jul 29 20:55:11.846325 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:11.846275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk" event={"ID":"f4e626f0-7f5c-4d5b-825e-383568363f1f","Type":"ContainerStarted","Data":"6d1e304c6070602cc26726501e19ebba81551e9292ed4dccdb4487cfbd37abeb"}
Jul 29 20:55:11.862195 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:11.862153    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7rcxk" podStartSLOduration=2.463267799 podStartE2EDuration="2.862137856s" totalImagesPullingTime="398.870057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:11.861433234 +0000 UTC m=+461.177922149" watchObservedRunningTime="2026-07-29 20:55:11.862137856 +0000 UTC m=+461.178626771"
Jul 29 20:55:16.848213 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:16.848181    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-54t75"
Jul 29 20:55:41.377842 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.377805    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"]
Jul 29 20:55:41.382380 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.382366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.384605 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.384584    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 29 20:55:41.384730 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.384592    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cpktn\""
Jul 29 20:55:41.385305 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.385279    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 29 20:55:41.389459 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.388802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"]
Jul 29 20:55:41.521567 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.521539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.521694 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.521584    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8m56\" (UniqueName: \"kubernetes.io/projected/61e1c420-486a-4bf8-9555-f7cb905fc79c-kube-api-access-n8m56\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.521694 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.521640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.622514 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.622485    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.622639 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.622523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8m56\" (UniqueName: \"kubernetes.io/projected/61e1c420-486a-4bf8-9555-f7cb905fc79c-kube-api-access-n8m56\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.622639 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.622546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.622896 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.622877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.622930 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.622921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.630253 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.630192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8m56\" (UniqueName: \"kubernetes.io/projected/61e1c420-486a-4bf8-9555-f7cb905fc79c-kube-api-access-n8m56\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.691782 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.691762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:41.829911 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.829882    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"]
Jul 29 20:55:41.833230 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:55:41.833204    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e1c420_486a_4bf8_9555_f7cb905fc79c.slice/crio-621c4e1e9211845bfdfc9a0728e8ffc5a172de66f71b87db309743b4b59f6542 WatchSource:0}: Error finding container 621c4e1e9211845bfdfc9a0728e8ffc5a172de66f71b87db309743b4b59f6542: Status 404 returned error can't find the container with id 621c4e1e9211845bfdfc9a0728e8ffc5a172de66f71b87db309743b4b59f6542
Jul 29 20:55:41.969798 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.969773    2579 generic.go:356] "Generic (PLEG): container finished" podID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerID="acaf2df96315ffbcc8222959e67799a0ee5622cdd689718ef8576d670eddac1d" exitCode=0
Jul 29 20:55:41.969916 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.969809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk" event={"ID":"61e1c420-486a-4bf8-9555-f7cb905fc79c","Type":"ContainerDied","Data":"acaf2df96315ffbcc8222959e67799a0ee5622cdd689718ef8576d670eddac1d"}
Jul 29 20:55:41.969916 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:41.969853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk" event={"ID":"61e1c420-486a-4bf8-9555-f7cb905fc79c","Type":"ContainerStarted","Data":"621c4e1e9211845bfdfc9a0728e8ffc5a172de66f71b87db309743b4b59f6542"}
Jul 29 20:55:42.976415 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:42.976341    2579 generic.go:356] "Generic (PLEG): container finished" podID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerID="c7f809351c4530cb0136e4827ba25750de9005af002c27516d54ae819703fd24" exitCode=0
Jul 29 20:55:42.976415 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:42.976387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk" event={"ID":"61e1c420-486a-4bf8-9555-f7cb905fc79c","Type":"ContainerDied","Data":"c7f809351c4530cb0136e4827ba25750de9005af002c27516d54ae819703fd24"}
Jul 29 20:55:43.982475 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:43.982439    2579 generic.go:356] "Generic (PLEG): container finished" podID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerID="9357ffd5f68f8cfb65a850fcc321a958ebd75b3ac6e87d9a7259930176076c0f" exitCode=0
Jul 29 20:55:43.983037 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:43.982524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk" event={"ID":"61e1c420-486a-4bf8-9555-f7cb905fc79c","Type":"ContainerDied","Data":"9357ffd5f68f8cfb65a850fcc321a958ebd75b3ac6e87d9a7259930176076c0f"}
Jul 29 20:55:45.101510 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.101489    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:45.250342 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.250269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-util\") pod \"61e1c420-486a-4bf8-9555-f7cb905fc79c\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") "
Jul 29 20:55:45.250489 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.250341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-bundle\") pod \"61e1c420-486a-4bf8-9555-f7cb905fc79c\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") "
Jul 29 20:55:45.250489 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.250412    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8m56\" (UniqueName: \"kubernetes.io/projected/61e1c420-486a-4bf8-9555-f7cb905fc79c-kube-api-access-n8m56\") pod \"61e1c420-486a-4bf8-9555-f7cb905fc79c\" (UID: \"61e1c420-486a-4bf8-9555-f7cb905fc79c\") "
Jul 29 20:55:45.251051 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.251017    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-bundle" (OuterVolumeSpecName: "bundle") pod "61e1c420-486a-4bf8-9555-f7cb905fc79c" (UID: "61e1c420-486a-4bf8-9555-f7cb905fc79c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:55:45.252404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.252372    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61e1c420-486a-4bf8-9555-f7cb905fc79c-kube-api-access-n8m56" (OuterVolumeSpecName: "kube-api-access-n8m56") pod "61e1c420-486a-4bf8-9555-f7cb905fc79c" (UID: "61e1c420-486a-4bf8-9555-f7cb905fc79c"). InnerVolumeSpecName "kube-api-access-n8m56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:55:45.255150 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.255130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-util" (OuterVolumeSpecName: "util") pod "61e1c420-486a-4bf8-9555-f7cb905fc79c" (UID: "61e1c420-486a-4bf8-9555-f7cb905fc79c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 20:55:45.351190 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.351169    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8m56\" (UniqueName: \"kubernetes.io/projected/61e1c420-486a-4bf8-9555-f7cb905fc79c-kube-api-access-n8m56\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:55:45.351190 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.351190    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-util\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:55:45.351331 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.351199    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61e1c420-486a-4bf8-9555-f7cb905fc79c-bundle\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:55:45.991865 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.991834    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk" event={"ID":"61e1c420-486a-4bf8-9555-f7cb905fc79c","Type":"ContainerDied","Data":"621c4e1e9211845bfdfc9a0728e8ffc5a172de66f71b87db309743b4b59f6542"}
Jul 29 20:55:45.992040 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.991881    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="621c4e1e9211845bfdfc9a0728e8ffc5a172de66f71b87db309743b4b59f6542"
Jul 29 20:55:45.992040 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:45.991843    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afv9npk"
Jul 29 20:55:49.872308 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872252    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"]
Jul 29 20:55:49.872693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872546    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerName="extract"
Jul 29 20:55:49.872693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872557    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerName="extract"
Jul 29 20:55:49.872693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872566    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerName="pull"
Jul 29 20:55:49.872693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872571    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerName="pull"
Jul 29 20:55:49.872693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872592    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerName="util"
Jul 29 20:55:49.872693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872598    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerName="util"
Jul 29 20:55:49.872693 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.872642    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e1c420-486a-4bf8-9555-f7cb905fc79c" containerName="extract"
Jul 29 20:55:49.874753 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.874738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:49.877743 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.877711    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 20:55:49.877890 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.877751    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 20:55:49.877890 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.877752    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-5qk7z\""
Jul 29 20:55:49.877890 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.877797    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 20:55:49.877890 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.877755    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 20:55:49.877890 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.877755    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 20:55:49.888871 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.888852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"]
Jul 29 20:55:49.986447 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.986416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84445ff9-de69-45a0-9c38-31c539dfd9b1-cert\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:49.986447 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.986448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dtr8\" (UniqueName: \"kubernetes.io/projected/84445ff9-de69-45a0-9c38-31c539dfd9b1-kube-api-access-5dtr8\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:49.986643 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.986473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/84445ff9-de69-45a0-9c38-31c539dfd9b1-manager-config\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:49.986643 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:49.986558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/84445ff9-de69-45a0-9c38-31c539dfd9b1-metrics-cert\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.087585 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.087540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84445ff9-de69-45a0-9c38-31c539dfd9b1-cert\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.087585 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.087587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dtr8\" (UniqueName: \"kubernetes.io/projected/84445ff9-de69-45a0-9c38-31c539dfd9b1-kube-api-access-5dtr8\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.087842 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.087608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/84445ff9-de69-45a0-9c38-31c539dfd9b1-manager-config\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.087842 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.087633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/84445ff9-de69-45a0-9c38-31c539dfd9b1-metrics-cert\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.088276 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.088255    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/84445ff9-de69-45a0-9c38-31c539dfd9b1-manager-config\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.089939 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.089921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/84445ff9-de69-45a0-9c38-31c539dfd9b1-metrics-cert\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.090029 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.090013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/84445ff9-de69-45a0-9c38-31c539dfd9b1-cert\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.100812 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.100793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dtr8\" (UniqueName: \"kubernetes.io/projected/84445ff9-de69-45a0-9c38-31c539dfd9b1-kube-api-access-5dtr8\") pod \"lws-controller-manager-64fb5f9c89-dg9k9\" (UID: \"84445ff9-de69-45a0-9c38-31c539dfd9b1\") " pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.184242 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.184167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:50.309090 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:50.309064    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"]
Jul 29 20:55:50.311674 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:55:50.311641    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84445ff9_de69_45a0_9c38_31c539dfd9b1.slice/crio-f683703e0dea30fb014b51c299edaf1ad46bc118aefa8e616a0b86e39910d61e WatchSource:0}: Error finding container f683703e0dea30fb014b51c299edaf1ad46bc118aefa8e616a0b86e39910d61e: Status 404 returned error can't find the container with id f683703e0dea30fb014b51c299edaf1ad46bc118aefa8e616a0b86e39910d61e
Jul 29 20:55:51.013197 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:51.013165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9" event={"ID":"84445ff9-de69-45a0-9c38-31c539dfd9b1","Type":"ContainerStarted","Data":"f683703e0dea30fb014b51c299edaf1ad46bc118aefa8e616a0b86e39910d61e"}
Jul 29 20:55:55.507117 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.507082    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"]
Jul 29 20:55:55.510334 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.510264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.512965 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.512850    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 20:55:55.513202 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.513179    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 20:55:55.513948 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.513547    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f44g6\""
Jul 29 20:55:55.513948 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.513783    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 20:55:55.513948 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.513839    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 20:55:55.526614 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.526592    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"]
Jul 29 20:55:55.627476 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.627443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ca56f15-86bc-4c57-9025-b0dbfde96e35-webhook-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.627648 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.627483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ca56f15-86bc-4c57-9025-b0dbfde96e35-manifests\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.627648 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.627512    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgsbz\" (UniqueName: \"kubernetes.io/projected/3ca56f15-86bc-4c57-9025-b0dbfde96e35-kube-api-access-dgsbz\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.627648 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.627542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ca56f15-86bc-4c57-9025-b0dbfde96e35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.627648 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.627578    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca56f15-86bc-4c57-9025-b0dbfde96e35-tmp\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.729021 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.728989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ca56f15-86bc-4c57-9025-b0dbfde96e35-webhook-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.729183 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.729028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ca56f15-86bc-4c57-9025-b0dbfde96e35-manifests\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.729240 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.729208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgsbz\" (UniqueName: \"kubernetes.io/projected/3ca56f15-86bc-4c57-9025-b0dbfde96e35-kube-api-access-dgsbz\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.729299 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.729255    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ca56f15-86bc-4c57-9025-b0dbfde96e35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.729362 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.729299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca56f15-86bc-4c57-9025-b0dbfde96e35-tmp\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.729414 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.729385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ca56f15-86bc-4c57-9025-b0dbfde96e35-manifests\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.731408 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.731385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ca56f15-86bc-4c57-9025-b0dbfde96e35-webhook-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.731550 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.731437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca56f15-86bc-4c57-9025-b0dbfde96e35-tmp\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.731659 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.731627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ca56f15-86bc-4c57-9025-b0dbfde96e35-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.738552 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.738529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgsbz\" (UniqueName: \"kubernetes.io/projected/3ca56f15-86bc-4c57-9025-b0dbfde96e35-kube-api-access-dgsbz\") pod \"opendatahub-operator-controller-manager-57645d8769-n9dbs\" (UID: \"3ca56f15-86bc-4c57-9025-b0dbfde96e35\") " pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:55.824824 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:55.824755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:55:56.782544 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:56.782516    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"]
Jul 29 20:55:56.784257 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:55:56.784233    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca56f15_86bc_4c57_9025_b0dbfde96e35.slice/crio-871a951906b85b5a0c5ae4a5d6e0280a3118dc6ef4f937dd24c15d95e3309ce7 WatchSource:0}: Error finding container 871a951906b85b5a0c5ae4a5d6e0280a3118dc6ef4f937dd24c15d95e3309ce7: Status 404 returned error can't find the container with id 871a951906b85b5a0c5ae4a5d6e0280a3118dc6ef4f937dd24c15d95e3309ce7
Jul 29 20:55:57.042567 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:57.042489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs" event={"ID":"3ca56f15-86bc-4c57-9025-b0dbfde96e35","Type":"ContainerStarted","Data":"871a951906b85b5a0c5ae4a5d6e0280a3118dc6ef4f937dd24c15d95e3309ce7"}
Jul 29 20:55:57.043846 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:57.043825    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9" event={"ID":"84445ff9-de69-45a0-9c38-31c539dfd9b1","Type":"ContainerStarted","Data":"c0fa25d56782a151cebb5f8782a8c34feab454dbab584d6e457839656bfde6c3"}
Jul 29 20:55:57.043961 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:57.043946    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:55:57.061703 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:55:57.061661    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9" podStartSLOduration=1.679687079 podStartE2EDuration="8.061646185s" totalImagesPullingTime="6.381959106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:55:57.060378578 +0000 UTC m=+506.376867488" watchObservedRunningTime="2026-07-29 20:55:57.061646185 +0000 UTC m=+506.378135099"
Jul 29 20:56:00.061780 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:00.061738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs" event={"ID":"3ca56f15-86bc-4c57-9025-b0dbfde96e35","Type":"ContainerStarted","Data":"3339b1693aac66fa522f54aad438e15a10e005ef53faedf8ab038c1d4f8d995b"}
Jul 29 20:56:01.067901 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:01.067864    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ca56f15-86bc-4c57-9025-b0dbfde96e35" containerID="3339b1693aac66fa522f54aad438e15a10e005ef53faedf8ab038c1d4f8d995b" exitCode=0
Jul 29 20:56:01.068254 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:01.067913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs" event={"ID":"3ca56f15-86bc-4c57-9025-b0dbfde96e35","Type":"ContainerDied","Data":"3339b1693aac66fa522f54aad438e15a10e005ef53faedf8ab038c1d4f8d995b"}
Jul 29 20:56:02.076081 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:02.076046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs" event={"ID":"3ca56f15-86bc-4c57-9025-b0dbfde96e35","Type":"ContainerStarted","Data":"b0a1b4c87f82a00defe2c7fdb90bccb74d8f137b60be3be6f88300394a0d5c74"}
Jul 29 20:56:02.076581 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:02.076186    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:56:02.099927 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:02.099868    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs" podStartSLOduration=2.914239721 podStartE2EDuration="7.099850562s" totalImagesPullingTime="3.180506691s" totalInitContainerRuntime="1.00510415s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:55:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:02.098358947 +0000 UTC m=+511.414847860" watchObservedRunningTime="2026-07-29 20:56:02.099850562 +0000 UTC m=+511.416339476"
Jul 29 20:56:08.052335 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:08.052306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-64fb5f9c89-dg9k9"
Jul 29 20:56:13.082993 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:13.082963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-57645d8769-n9dbs"
Jul 29 20:56:15.417373 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.417336    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"]
Jul 29 20:56:15.434618 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.434590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"]
Jul 29 20:56:15.434749 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.434684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.439017 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.438996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 29 20:56:15.439184 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.439018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 29 20:56:15.439561 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.439539    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7g5vs\""
Jul 29 20:56:15.439997 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.439983    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 20:56:15.440757 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.440737    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 20:56:15.475809 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.475787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3181-bfa7-4674-9428-a23fba762430-tmp\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.475930 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.475828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3181-bfa7-4674-9428-a23fba762430-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.475930 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.475893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5zs\" (UniqueName: \"kubernetes.io/projected/643b3181-bfa7-4674-9428-a23fba762430-kube-api-access-gj5zs\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.576666 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.576639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3181-bfa7-4674-9428-a23fba762430-tmp\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.576804 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.576685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3181-bfa7-4674-9428-a23fba762430-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.576804 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.576712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5zs\" (UniqueName: \"kubernetes.io/projected/643b3181-bfa7-4674-9428-a23fba762430-kube-api-access-gj5zs\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.576804 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:56:15.576799    2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 29 20:56:15.576979 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:56:15.576870    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/643b3181-bfa7-4674-9428-a23fba762430-tls-certs podName:643b3181-bfa7-4674-9428-a23fba762430 nodeName:}" failed. No retries permitted until 2026-07-29 20:56:16.07685449 +0000 UTC m=+525.393343407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/643b3181-bfa7-4674-9428-a23fba762430-tls-certs") pod "kube-auth-proxy-68db9f4dd4-wjm28" (UID: "643b3181-bfa7-4674-9428-a23fba762430") : secret "kube-auth-proxy-tls" not found
Jul 29 20:56:15.578843 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.578820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/643b3181-bfa7-4674-9428-a23fba762430-tmp\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:15.591851 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:15.591823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5zs\" (UniqueName: \"kubernetes.io/projected/643b3181-bfa7-4674-9428-a23fba762430-kube-api-access-gj5zs\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:16.080003 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:16.079971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3181-bfa7-4674-9428-a23fba762430-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:16.082299 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:16.082259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/643b3181-bfa7-4674-9428-a23fba762430-tls-certs\") pod \"kube-auth-proxy-68db9f4dd4-wjm28\" (UID: \"643b3181-bfa7-4674-9428-a23fba762430\") " pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:16.345281 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:16.345251    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"
Jul 29 20:56:16.485850 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:16.485825    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28"]
Jul 29 20:56:16.488887 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:56:16.488858    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643b3181_bfa7_4674_9428_a23fba762430.slice/crio-536076e6ac4dc5312e6df4b757b92bcb3ab1ee6da9cb6f85ff780bbafcab2e94 WatchSource:0}: Error finding container 536076e6ac4dc5312e6df4b757b92bcb3ab1ee6da9cb6f85ff780bbafcab2e94: Status 404 returned error can't find the container with id 536076e6ac4dc5312e6df4b757b92bcb3ab1ee6da9cb6f85ff780bbafcab2e94
Jul 29 20:56:17.145299 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:17.145254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28" event={"ID":"643b3181-bfa7-4674-9428-a23fba762430","Type":"ContainerStarted","Data":"536076e6ac4dc5312e6df4b757b92bcb3ab1ee6da9cb6f85ff780bbafcab2e94"}
Jul 29 20:56:19.158379 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:19.158316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28" event={"ID":"643b3181-bfa7-4674-9428-a23fba762430","Type":"ContainerStarted","Data":"a46a1c3045ca9efb7c0b8fc1e19fa65558d62ab6dfce836a4c50d16c8100a622"}
Jul 29 20:56:19.174151 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:56:19.174098    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68db9f4dd4-wjm28" podStartSLOduration=1.882688893 podStartE2EDuration="4.17408076s" totalImagesPullingTime="2.291391867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:56:19.173453151 +0000 UTC m=+528.489942082" watchObservedRunningTime="2026-07-29 20:56:19.17408076 +0000 UTC m=+528.490569676"
Jul 29 20:57:45.802280 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:57:45.802248    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b7bfbb774-wzqgn"]
Jul 29 20:58:03.613109 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.613029    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"]
Jul 29 20:58:03.615567 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.615545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.617945 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.617913    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 29 20:58:03.617945 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.617927    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 29 20:58:03.618102 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.617948    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hlkjq\""
Jul 29 20:58:03.618497 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.618481    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 20:58:03.618604 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.618481    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 20:58:03.626224 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.626205    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"]
Jul 29 20:58:03.717233 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.717204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.717233 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.717238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.717452 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.717267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwv9\" (UniqueName: \"kubernetes.io/projected/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-kube-api-access-jvwv9\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.818535 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.818507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.818679 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.818541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.818679 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.818570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwv9\" (UniqueName: \"kubernetes.io/projected/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-kube-api-access-jvwv9\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.818679 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:58:03.818653    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 29 20:58:03.818857 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:58:03.818735    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-plugin-serving-cert podName:048aaed8-8b03-40e2-ab2a-7fa6f2e62f99 nodeName:}" failed. No retries permitted until 2026-07-29 20:58:04.318712681 +0000 UTC m=+633.635201575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-7rqvx" (UID: "048aaed8-8b03-40e2-ab2a-7fa6f2e62f99") : secret "plugin-serving-cert" not found
Jul 29 20:58:03.819168 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.819151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:03.835755 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:03.835736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwv9\" (UniqueName: \"kubernetes.io/projected/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-kube-api-access-jvwv9\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:04.323109 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:04.323080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:04.325404 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:04.325379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/048aaed8-8b03-40e2-ab2a-7fa6f2e62f99-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-7rqvx\" (UID: \"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:04.524095 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:04.524071    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"
Jul 29 20:58:04.652195 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:04.652173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx"]
Jul 29 20:58:04.654575 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:58:04.654551    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod048aaed8_8b03_40e2_ab2a_7fa6f2e62f99.slice/crio-0348aada3f1ddb832b4717be41529870aeb3d5bd59303a4d609a18d849a206d1 WatchSource:0}: Error finding container 0348aada3f1ddb832b4717be41529870aeb3d5bd59303a4d609a18d849a206d1: Status 404 returned error can't find the container with id 0348aada3f1ddb832b4717be41529870aeb3d5bd59303a4d609a18d849a206d1
Jul 29 20:58:05.631494 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:05.631455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx" event={"ID":"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99","Type":"ContainerStarted","Data":"0348aada3f1ddb832b4717be41529870aeb3d5bd59303a4d609a18d849a206d1"}
Jul 29 20:58:10.825834 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:10.825771    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b7bfbb774-wzqgn" podUID="a80c20c9-f60c-4d7c-a2a9-73e437946295" containerName="console" containerID="cri-o://7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48" gracePeriod=15
Jul 29 20:58:11.094587 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.094559    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:58:11.180836 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.180807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-oauth-serving-cert\") pod \"a80c20c9-f60c-4d7c-a2a9-73e437946295\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") "
Jul 29 20:58:11.180985 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.180842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvqvx\" (UniqueName: \"kubernetes.io/projected/a80c20c9-f60c-4d7c-a2a9-73e437946295-kube-api-access-mvqvx\") pod \"a80c20c9-f60c-4d7c-a2a9-73e437946295\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") "
Jul 29 20:58:11.180985 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.180863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-serving-cert\") pod \"a80c20c9-f60c-4d7c-a2a9-73e437946295\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") "
Jul 29 20:58:11.180985 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.180879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-config\") pod \"a80c20c9-f60c-4d7c-a2a9-73e437946295\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") "
Jul 29 20:58:11.181131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181009    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-oauth-config\") pod \"a80c20c9-f60c-4d7c-a2a9-73e437946295\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") "
Jul 29 20:58:11.181131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181049    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-trusted-ca-bundle\") pod \"a80c20c9-f60c-4d7c-a2a9-73e437946295\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") "
Jul 29 20:58:11.181131 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181088    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-service-ca\") pod \"a80c20c9-f60c-4d7c-a2a9-73e437946295\" (UID: \"a80c20c9-f60c-4d7c-a2a9-73e437946295\") "
Jul 29 20:58:11.181345 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181321    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-config" (OuterVolumeSpecName: "console-config") pod "a80c20c9-f60c-4d7c-a2a9-73e437946295" (UID: "a80c20c9-f60c-4d7c-a2a9-73e437946295"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:58:11.181432 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181412    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-config\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:58:11.181492 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181385    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a80c20c9-f60c-4d7c-a2a9-73e437946295" (UID: "a80c20c9-f60c-4d7c-a2a9-73e437946295"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:58:11.181552 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a80c20c9-f60c-4d7c-a2a9-73e437946295" (UID: "a80c20c9-f60c-4d7c-a2a9-73e437946295"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:58:11.181602 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.181547    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-service-ca" (OuterVolumeSpecName: "service-ca") pod "a80c20c9-f60c-4d7c-a2a9-73e437946295" (UID: "a80c20c9-f60c-4d7c-a2a9-73e437946295"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 20:58:11.183108 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.183086    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a80c20c9-f60c-4d7c-a2a9-73e437946295" (UID: "a80c20c9-f60c-4d7c-a2a9-73e437946295"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:58:11.183436 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.183418    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a80c20c9-f60c-4d7c-a2a9-73e437946295" (UID: "a80c20c9-f60c-4d7c-a2a9-73e437946295"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 20:58:11.183492 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.183453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80c20c9-f60c-4d7c-a2a9-73e437946295-kube-api-access-mvqvx" (OuterVolumeSpecName: "kube-api-access-mvqvx") pod "a80c20c9-f60c-4d7c-a2a9-73e437946295" (UID: "a80c20c9-f60c-4d7c-a2a9-73e437946295"). InnerVolumeSpecName "kube-api-access-mvqvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 20:58:11.281975 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.281948    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-oauth-config\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:58:11.281975 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.281974    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-trusted-ca-bundle\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:58:11.282142 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.281985    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:58:11.282142 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.281994    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a80c20c9-f60c-4d7c-a2a9-73e437946295-oauth-serving-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:58:11.282142 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.282006    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvqvx\" (UniqueName: \"kubernetes.io/projected/a80c20c9-f60c-4d7c-a2a9-73e437946295-kube-api-access-mvqvx\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:58:11.282142 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.282020    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a80c20c9-f60c-4d7c-a2a9-73e437946295-console-serving-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 20:58:11.669141 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.669100    2579 generic.go:356] "Generic (PLEG): container finished" podID="a80c20c9-f60c-4d7c-a2a9-73e437946295" containerID="7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48" exitCode=0
Jul 29 20:58:11.669344 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.669167    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b7bfbb774-wzqgn"
Jul 29 20:58:11.669344 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.669182    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7bfbb774-wzqgn" event={"ID":"a80c20c9-f60c-4d7c-a2a9-73e437946295","Type":"ContainerDied","Data":"7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48"}
Jul 29 20:58:11.669344 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.669241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b7bfbb774-wzqgn" event={"ID":"a80c20c9-f60c-4d7c-a2a9-73e437946295","Type":"ContainerDied","Data":"8211b005288661868516d8ec7dd0fab46e2c816eeebc221ffc483e2feac03e59"}
Jul 29 20:58:11.669344 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.669268    2579 scope.go:117] "RemoveContainer" containerID="7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48"
Jul 29 20:58:11.687975 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.687952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b7bfbb774-wzqgn"]
Jul 29 20:58:11.694308 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:11.693368    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b7bfbb774-wzqgn"]
Jul 29 20:58:13.256211 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.256173    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a80c20c9-f60c-4d7c-a2a9-73e437946295" path="/var/lib/kubelet/pods/a80c20c9-f60c-4d7c-a2a9-73e437946295/volumes"
Jul 29 20:58:13.750829 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.750795    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"]
Jul 29 20:58:13.751242 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.751222    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80c20c9-f60c-4d7c-a2a9-73e437946295" containerName="console"
Jul 29 20:58:13.751242 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.751242    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80c20c9-f60c-4d7c-a2a9-73e437946295" containerName="console"
Jul 29 20:58:13.751424 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.751360    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a80c20c9-f60c-4d7c-a2a9-73e437946295" containerName="console"
Jul 29 20:58:13.757015 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.756909    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"
Jul 29 20:58:13.760209 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.760094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qz554\""
Jul 29 20:58:13.771921 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.771899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"]
Jul 29 20:58:13.906944 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:13.906900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgwn9\" (UniqueName: \"kubernetes.io/projected/4de8b85f-818f-4e3e-8bb1-fb19b637b0d6-kube-api-access-hgwn9\") pod \"limitador-operator-controller-manager-859d7b78bd-lkksg\" (UID: \"4de8b85f-818f-4e3e-8bb1-fb19b637b0d6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"
Jul 29 20:58:14.007939 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:14.007858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgwn9\" (UniqueName: \"kubernetes.io/projected/4de8b85f-818f-4e3e-8bb1-fb19b637b0d6-kube-api-access-hgwn9\") pod \"limitador-operator-controller-manager-859d7b78bd-lkksg\" (UID: \"4de8b85f-818f-4e3e-8bb1-fb19b637b0d6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"
Jul 29 20:58:14.020524 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:14.020497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgwn9\" (UniqueName: \"kubernetes.io/projected/4de8b85f-818f-4e3e-8bb1-fb19b637b0d6-kube-api-access-hgwn9\") pod \"limitador-operator-controller-manager-859d7b78bd-lkksg\" (UID: \"4de8b85f-818f-4e3e-8bb1-fb19b637b0d6\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"
Jul 29 20:58:14.070548 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:14.070513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"
Jul 29 20:58:25.293378 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:25.292982    2579 scope.go:117] "RemoveContainer" containerID="7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48"
Jul 29 20:58:25.293378 ip-10-0-136-64 kubenswrapper[2579]: E0729 20:58:25.293358    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48\": container with ID starting with 7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48 not found: ID does not exist" containerID="7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48"
Jul 29 20:58:25.293716 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:25.293397    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48"} err="failed to get container status \"7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48\": rpc error: code = NotFound desc = could not find container \"7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48\": container with ID starting with 7db8f1049163b51ae74e3cd433603fe43c1d24f2afab46d84e4aea74e8840b48 not found: ID does not exist"
Jul 29 20:58:25.424783 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:25.424755    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"]
Jul 29 20:58:25.427666 ip-10-0-136-64 kubenswrapper[2579]: W0729 20:58:25.427635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de8b85f_818f_4e3e_8bb1_fb19b637b0d6.slice/crio-3cddb782041665e1cce2eafe6f5bdfbc00c470dca4fd54d55071f968f50c86b0 WatchSource:0}: Error finding container 3cddb782041665e1cce2eafe6f5bdfbc00c470dca4fd54d55071f968f50c86b0: Status 404 returned error can't find the container with id 3cddb782041665e1cce2eafe6f5bdfbc00c470dca4fd54d55071f968f50c86b0
Jul 29 20:58:25.741791 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:25.741760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx" event={"ID":"048aaed8-8b03-40e2-ab2a-7fa6f2e62f99","Type":"ContainerStarted","Data":"e2a2e0235d6b9ed4e29c868a764cbee4f632e88d813cc276d465f4bf65aceab3"}
Jul 29 20:58:25.742844 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:25.742822    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg" event={"ID":"4de8b85f-818f-4e3e-8bb1-fb19b637b0d6","Type":"ContainerStarted","Data":"3cddb782041665e1cce2eafe6f5bdfbc00c470dca4fd54d55071f968f50c86b0"}
Jul 29 20:58:25.757334 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:25.757253    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-7rqvx" podStartSLOduration=2.064915944 podStartE2EDuration="22.757240655s" totalImagesPullingTime="20.692324711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:25.756532194 +0000 UTC m=+655.073021125" watchObservedRunningTime="2026-07-29 20:58:25.757240655 +0000 UTC m=+655.073729569"
Jul 29 20:58:27.752123 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:27.752089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg" event={"ID":"4de8b85f-818f-4e3e-8bb1-fb19b637b0d6","Type":"ContainerStarted","Data":"ff455a13fd4ba0ec934fc3ac6719962ecd04f0eb8980ddb7dbc3fbf2092540bd"}
Jul 29 20:58:27.752539 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:27.752190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"
Jul 29 20:58:27.772447 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:27.772389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg" podStartSLOduration=13.065254295 podStartE2EDuration="14.772375992s" totalImagesPullingTime="1.707121697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 20:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 20:58:27.77193021 +0000 UTC m=+657.088419136" watchObservedRunningTime="2026-07-29 20:58:27.772375992 +0000 UTC m=+657.088864950"
Jul 29 20:58:38.759371 ip-10-0-136-64 kubenswrapper[2579]: I0729 20:58:38.759333    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkksg"
Jul 29 21:00:46.032172 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.032135    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"]
Jul 29 21:00:46.034762 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.034742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.038003 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.037981    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 21:00:46.038003 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.037989    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 21:00:46.038202 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.038033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 29 21:00:46.038202 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.037989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xm4qm\""
Jul 29 21:00:46.044461 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.044439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"]
Jul 29 21:00:46.129514 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.129486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.129632 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.129522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmnjr\" (UniqueName: \"kubernetes.io/projected/aad62943-cb5a-4e0d-af69-900892198939-kube-api-access-gmnjr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.129632 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.129551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.129632 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.129602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.129755 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.129635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad62943-cb5a-4e0d-af69-900892198939-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.129755 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.129653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230318 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230420 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230420 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad62943-cb5a-4e0d-af69-900892198939-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230420 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230546 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230546 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmnjr\" (UniqueName: \"kubernetes.io/projected/aad62943-cb5a-4e0d-af69-900892198939-kube-api-access-gmnjr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230681 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230660    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230748 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.230748 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.230722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.232580 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.232562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aad62943-cb5a-4e0d-af69-900892198939-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.232722 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.232706    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aad62943-cb5a-4e0d-af69-900892198939-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.237580 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.237559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmnjr\" (UniqueName: \"kubernetes.io/projected/aad62943-cb5a-4e0d-af69-900892198939-kube-api-access-gmnjr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-gstzs\" (UID: \"aad62943-cb5a-4e0d-af69-900892198939\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.345752 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.345731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:46.473365 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.473338    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"]
Jul 29 21:00:46.475387 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:00:46.475343    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad62943_cb5a_4e0d_af69_900892198939.slice/crio-8db73a62fee6b6e2dc39e06a12185f70dc243734cd7866ad376bf1104d8afeab WatchSource:0}: Error finding container 8db73a62fee6b6e2dc39e06a12185f70dc243734cd7866ad376bf1104d8afeab: Status 404 returned error can't find the container with id 8db73a62fee6b6e2dc39e06a12185f70dc243734cd7866ad376bf1104d8afeab
Jul 29 21:00:46.476761 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:46.476739    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:00:47.382379 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:47.382349    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs" event={"ID":"aad62943-cb5a-4e0d-af69-900892198939","Type":"ContainerStarted","Data":"8db73a62fee6b6e2dc39e06a12185f70dc243734cd7866ad376bf1104d8afeab"}
Jul 29 21:00:53.410714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:53.410680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs" event={"ID":"aad62943-cb5a-4e0d-af69-900892198939","Type":"ContainerStarted","Data":"9d5d920153dcd37cc27461ef4ed6a5bedac636ea9398669d311a506c37242dc4"}
Jul 29 21:00:53.430633 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:53.430585    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs" podStartSLOduration=0.693560805 podStartE2EDuration="7.43056752s" totalImagesPullingTime="6.737006715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:00:53.429533165 +0000 UTC m=+802.746022081" watchObservedRunningTime="2026-07-29 21:00:53.43056752 +0000 UTC m=+802.747056438"
Jul 29 21:00:56.346182 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:56.346146    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:56.346668 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:56.346206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:56.358629 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:56.358599    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:00:56.441325 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:00:56.441301    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-gstzs"
Jul 29 21:01:13.956532 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:13.956499    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"]
Jul 29 21:01:13.959619 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:13.959603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:13.961770 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:13.961750    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 29 21:01:13.968324 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:13.968304    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"]
Jul 29 21:01:14.145762 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.145729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1371ff-4afa-469b-be32-f021cb7f8a70-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.145921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.145774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.145921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.145796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.145921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.145813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.145921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.145859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjff\" (UniqueName: \"kubernetes.io/projected/fc1371ff-4afa-469b-be32-f021cb7f8a70-kube-api-access-6hjff\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.145921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.145887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246525 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjff\" (UniqueName: \"kubernetes.io/projected/fc1371ff-4afa-469b-be32-f021cb7f8a70-kube-api-access-6hjff\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246525 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246678 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1371ff-4afa-469b-be32-f021cb7f8a70-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246678 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246554    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246678 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246678 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246871 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246928 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.246989 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.246971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.248832 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.248811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1371ff-4afa-469b-be32-f021cb7f8a70-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.249071 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.249055    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1371ff-4afa-469b-be32-f021cb7f8a70-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.253753 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.253736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjff\" (UniqueName: \"kubernetes.io/projected/fc1371ff-4afa-469b-be32-f021cb7f8a70-kube-api-access-6hjff\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86\" (UID: \"fc1371ff-4afa-469b-be32-f021cb7f8a70\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.269763 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.269743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:14.394711 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.394688    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"]
Jul 29 21:01:14.396485 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:01:14.396460    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1371ff_4afa_469b_be32_f021cb7f8a70.slice/crio-4ada63ddae14d3d43bb467765fd3294c40c8f3fd4fe7bf2a8ab75b2c7c54d83b WatchSource:0}: Error finding container 4ada63ddae14d3d43bb467765fd3294c40c8f3fd4fe7bf2a8ab75b2c7c54d83b: Status 404 returned error can't find the container with id 4ada63ddae14d3d43bb467765fd3294c40c8f3fd4fe7bf2a8ab75b2c7c54d83b
Jul 29 21:01:14.507382 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:14.507318    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86" event={"ID":"fc1371ff-4afa-469b-be32-f021cb7f8a70","Type":"ContainerStarted","Data":"4ada63ddae14d3d43bb467765fd3294c40c8f3fd4fe7bf2a8ab75b2c7c54d83b"}
Jul 29 21:01:15.513453 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:15.513421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86" event={"ID":"fc1371ff-4afa-469b-be32-f021cb7f8a70","Type":"ContainerStarted","Data":"ed07663ccf0f9ab0ad772e9dbdca40fa55f08666210118cad3cf89f5dfd12285"}
Jul 29 21:01:15.530209 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:15.530166    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86" podStartSLOduration=2.349641216 podStartE2EDuration="2.530151553s" totalImagesPullingTime="180.510337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:01:15.529143273 +0000 UTC m=+824.845632217" watchObservedRunningTime="2026-07-29 21:01:15.530151553 +0000 UTC m=+824.846640488"
Jul 29 21:01:24.270386 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:24.270351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:24.270857 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:24.270436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:24.283001 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:24.282982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:24.564384 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:24.564311    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86"
Jul 29 21:01:26.848943 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.848909    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"]
Jul 29 21:01:26.852091 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.852077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:26.854374 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.854354    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 29 21:01:26.862153 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.862135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"]
Jul 29 21:01:26.937326 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.937283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nckg\" (UniqueName: \"kubernetes.io/projected/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-kube-api-access-9nckg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:26.937450 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.937334    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:26.937450 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.937361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:26.937450 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.937403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:26.937450 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.937433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:26.937450 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:26.937449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038348 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nckg\" (UniqueName: \"kubernetes.io/projected/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-kube-api-access-9nckg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038685 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038739 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038686    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038793 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.038849 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.038810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.040787 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.040767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.040897 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.040880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.045503 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.045481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nckg\" (UniqueName: \"kubernetes.io/projected/a5f09ebf-353b-41eb-b1e9-af28d67a43b1-kube-api-access-9nckg\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb\" (UID: \"a5f09ebf-353b-41eb-b1e9-af28d67a43b1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.161806 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.161731    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:27.290724 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.290704    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"]
Jul 29 21:01:27.292990 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:01:27.292957    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f09ebf_353b_41eb_b1e9_af28d67a43b1.slice/crio-63678637c556ddfadef1bf68d3201eb47ac3929fd5b08c72a5432822af857c8e WatchSource:0}: Error finding container 63678637c556ddfadef1bf68d3201eb47ac3929fd5b08c72a5432822af857c8e: Status 404 returned error can't find the container with id 63678637c556ddfadef1bf68d3201eb47ac3929fd5b08c72a5432822af857c8e
Jul 29 21:01:27.569058 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:27.569032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb" event={"ID":"a5f09ebf-353b-41eb-b1e9-af28d67a43b1","Type":"ContainerStarted","Data":"63678637c556ddfadef1bf68d3201eb47ac3929fd5b08c72a5432822af857c8e"}
Jul 29 21:01:28.575359 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:28.575326    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb" event={"ID":"a5f09ebf-353b-41eb-b1e9-af28d67a43b1","Type":"ContainerStarted","Data":"955bc05dc4df7547bc846bd2989c343412c693f92329387ef791f7ec77b1e3a8"}
Jul 29 21:01:28.592703 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:28.592649    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb" podStartSLOduration=2.341439195 podStartE2EDuration="2.592635626s" totalImagesPullingTime="251.196431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:01:28.590426275 +0000 UTC m=+837.906915192" watchObservedRunningTime="2026-07-29 21:01:28.592635626 +0000 UTC m=+837.909124541"
Jul 29 21:01:37.162768 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:37.162728    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:37.163191 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:37.162787    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:37.175179 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:37.175154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:01:37.628537 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:01:37.628505    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb"
Jul 29 21:13:35.058498 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.058468    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"]
Jul 29 21:13:35.061715 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.061700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.064443 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.064427    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6947d68e-kserve-self-signed-certs\""
Jul 29 21:13:35.095145 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.095119    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"]
Jul 29 21:13:35.152408 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.152380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-tmp-dir\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.152545 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.152423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-dshm\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.152545 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.152447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7pkz\" (UniqueName: \"kubernetes.io/projected/b189937e-9131-4261-8ebb-bccfa463773e-kube-api-access-l7pkz\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.152545 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.152527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b189937e-9131-4261-8ebb-bccfa463773e-tls-certs\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.152696 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.152568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-model-cache\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.152696 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.152597    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-home\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.253899 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.253872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-tmp-dir\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254051 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.253910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-dshm\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254051 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.253927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7pkz\" (UniqueName: \"kubernetes.io/projected/b189937e-9131-4261-8ebb-bccfa463773e-kube-api-access-l7pkz\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254051 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.253966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b189937e-9131-4261-8ebb-bccfa463773e-tls-certs\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254051 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.253991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-model-cache\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254051 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.254007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-home\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254506 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.254478    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-home\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254506 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.254489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-tmp-dir\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.254695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.254517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-model-cache\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.256310 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.256269    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-dshm\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.256490 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.256469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b189937e-9131-4261-8ebb-bccfa463773e-tls-certs\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.264994 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.264971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7pkz\" (UniqueName: \"kubernetes.io/projected/b189937e-9131-4261-8ebb-bccfa463773e-kube-api-access-l7pkz\") pod \"e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") " pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.370053 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.370019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:35.499251 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.499141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"]
Jul 29 21:13:35.501967 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:13:35.501941    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb189937e_9131_4261_8ebb_bccfa463773e.slice/crio-9d291fbb3e9b8d756bc5693a4abe3fea5779432029b2d6d39957ec496cd08e20 WatchSource:0}: Error finding container 9d291fbb3e9b8d756bc5693a4abe3fea5779432029b2d6d39957ec496cd08e20: Status 404 returned error can't find the container with id 9d291fbb3e9b8d756bc5693a4abe3fea5779432029b2d6d39957ec496cd08e20
Jul 29 21:13:35.928981 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.928950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8" event={"ID":"b189937e-9131-4261-8ebb-bccfa463773e","Type":"ContainerStarted","Data":"9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d"}
Jul 29 21:13:35.928981 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.928986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8" event={"ID":"b189937e-9131-4261-8ebb-bccfa463773e","Type":"ContainerStarted","Data":"9d291fbb3e9b8d756bc5693a4abe3fea5779432029b2d6d39957ec496cd08e20"}
Jul 29 21:13:35.946532 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:35.946491    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8" podStartSLOduration=0.946476211 podStartE2EDuration="946.476211ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:35.944989854 +0000 UTC m=+1565.261478767" watchObservedRunningTime="2026-07-29 21:13:35.946476211 +0000 UTC m=+1565.262965125"
Jul 29 21:13:40.852699 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:40.852661    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"]
Jul 29 21:13:40.856262 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:40.856243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:40.858574 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:40.858556    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-6947d68e-kserve-self-signed-certs\""
Jul 29 21:13:40.864374 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:40.864352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"]
Jul 29 21:13:41.006068 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.006036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tmp-dir\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.006068 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.006074    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-home\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.006310 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.006120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5ng\" (UniqueName: \"kubernetes.io/projected/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-kube-api-access-mn5ng\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.006310 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.006186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-dshm\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.006310 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.006217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-model-cache\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.006310 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.006243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tls-certs\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.106988 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.106887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tmp-dir\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.106988 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.106940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-home\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.106988 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.106982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5ng\" (UniqueName: \"kubernetes.io/projected/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-kube-api-access-mn5ng\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.107274 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.107008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-dshm\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.107274 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.107048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-model-cache\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.107274 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.107082    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tls-certs\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.107476 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.107424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tmp-dir\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.107476 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.107437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-home\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.107548 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.107489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-model-cache\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.109360 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.109333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-dshm\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.109585 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.109569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tls-certs\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.114500 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.114479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5ng\" (UniqueName: \"kubernetes.io/projected/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-kube-api-access-mn5ng\") pod \"e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") " pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.166326 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.166283    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:41.296463 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.296441    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"]
Jul 29 21:13:41.298742 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:13:41.298717    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00f3039e_8bef_42e0_8ec2_90941c9ec9e6.slice/crio-15d7306b3fb3ed27aa7045772562162e1a0bc92351e9f4bcfda227aabb97545a WatchSource:0}: Error finding container 15d7306b3fb3ed27aa7045772562162e1a0bc92351e9f4bcfda227aabb97545a: Status 404 returned error can't find the container with id 15d7306b3fb3ed27aa7045772562162e1a0bc92351e9f4bcfda227aabb97545a
Jul 29 21:13:41.959765 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.959734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw" event={"ID":"00f3039e-8bef-42e0-8ec2-90941c9ec9e6","Type":"ContainerStarted","Data":"ee9d25cefb8dee3e04cbb19b8fe801a926eb485bdab2dabd5be6d5e09703551c"}
Jul 29 21:13:41.959765 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.959771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw" event={"ID":"00f3039e-8bef-42e0-8ec2-90941c9ec9e6","Type":"ContainerStarted","Data":"15d7306b3fb3ed27aa7045772562162e1a0bc92351e9f4bcfda227aabb97545a"}
Jul 29 21:13:41.977563 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:41.977512    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw" podStartSLOduration=1.9774945000000002 podStartE2EDuration="1.9774945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:13:41.976915511 +0000 UTC m=+1571.293404428" watchObservedRunningTime="2026-07-29 21:13:41.9774945 +0000 UTC m=+1571.293983420"
Jul 29 21:13:45.371011 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:45.370978    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:45.371398 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:45.371055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:45.383253 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:45.383234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:45.989859 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:45.989831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:13:51.166871 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:51.166833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:51.167258 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:51.166891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:51.179497 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:51.179474    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:13:52.017924 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:13:52.017888    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:14:13.411841 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:13.411810    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"]
Jul 29 21:14:14.968255 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:14.968225    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"]
Jul 29 21:14:28.429844 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.429745    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8" podUID="b189937e-9131-4261-8ebb-bccfa463773e" containerName="main" containerID="cri-o://9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d" gracePeriod=45
Jul 29 21:14:28.675638 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.675616    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:14:28.776155 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-tmp-dir\") pod \"b189937e-9131-4261-8ebb-bccfa463773e\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") "
Jul 29 21:14:28.776155 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7pkz\" (UniqueName: \"kubernetes.io/projected/b189937e-9131-4261-8ebb-bccfa463773e-kube-api-access-l7pkz\") pod \"b189937e-9131-4261-8ebb-bccfa463773e\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") "
Jul 29 21:14:28.776155 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776151    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-dshm\") pod \"b189937e-9131-4261-8ebb-bccfa463773e\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") "
Jul 29 21:14:28.776381 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b189937e-9131-4261-8ebb-bccfa463773e-tls-certs\") pod \"b189937e-9131-4261-8ebb-bccfa463773e\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") "
Jul 29 21:14:28.776381 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-model-cache\") pod \"b189937e-9131-4261-8ebb-bccfa463773e\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") "
Jul 29 21:14:28.776494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-home\") pod \"b189937e-9131-4261-8ebb-bccfa463773e\" (UID: \"b189937e-9131-4261-8ebb-bccfa463773e\") "
Jul 29 21:14:28.776494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776421    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b189937e-9131-4261-8ebb-bccfa463773e" (UID: "b189937e-9131-4261-8ebb-bccfa463773e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:28.776618 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776587    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b189937e-9131-4261-8ebb-bccfa463773e" (UID: "b189937e-9131-4261-8ebb-bccfa463773e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:28.776732 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776716    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:28.776781 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-home" (OuterVolumeSpecName: "home") pod "b189937e-9131-4261-8ebb-bccfa463773e" (UID: "b189937e-9131-4261-8ebb-bccfa463773e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:28.776781 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.776737    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:28.778220 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.778204    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b189937e-9131-4261-8ebb-bccfa463773e-kube-api-access-l7pkz" (OuterVolumeSpecName: "kube-api-access-l7pkz") pod "b189937e-9131-4261-8ebb-bccfa463773e" (UID: "b189937e-9131-4261-8ebb-bccfa463773e"). InnerVolumeSpecName "kube-api-access-l7pkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:28.778577 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.778561    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b189937e-9131-4261-8ebb-bccfa463773e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b189937e-9131-4261-8ebb-bccfa463773e" (UID: "b189937e-9131-4261-8ebb-bccfa463773e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:14:28.778652 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.778573    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-dshm" (OuterVolumeSpecName: "dshm") pod "b189937e-9131-4261-8ebb-bccfa463773e" (UID: "b189937e-9131-4261-8ebb-bccfa463773e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:28.877547 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.877527    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:28.877547 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.877545    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7pkz\" (UniqueName: \"kubernetes.io/projected/b189937e-9131-4261-8ebb-bccfa463773e-kube-api-access-l7pkz\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:28.877682 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.877555    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b189937e-9131-4261-8ebb-bccfa463773e-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:28.877682 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:28.877563    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b189937e-9131-4261-8ebb-bccfa463773e-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:29.193201 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.193171    2579 generic.go:356] "Generic (PLEG): container finished" podID="b189937e-9131-4261-8ebb-bccfa463773e" containerID="9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d" exitCode=0
Jul 29 21:14:29.193379 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.193221    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"
Jul 29 21:14:29.193379 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.193227    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8" event={"ID":"b189937e-9131-4261-8ebb-bccfa463773e","Type":"ContainerDied","Data":"9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d"}
Jul 29 21:14:29.193379 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.193256    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8" event={"ID":"b189937e-9131-4261-8ebb-bccfa463773e","Type":"ContainerDied","Data":"9d291fbb3e9b8d756bc5693a4abe3fea5779432029b2d6d39957ec496cd08e20"}
Jul 29 21:14:29.193379 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.193271    2579 scope.go:117] "RemoveContainer" containerID="9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d"
Jul 29 21:14:29.202000 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.201981    2579 scope.go:117] "RemoveContainer" containerID="9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d"
Jul 29 21:14:29.202247 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:14:29.202231    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d\": container with ID starting with 9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d not found: ID does not exist" containerID="9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d"
Jul 29 21:14:29.202317 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.202254    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d"} err="failed to get container status \"9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d\": rpc error: code = NotFound desc = could not find container \"9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d\": container with ID starting with 9898a6b6539b7611424e965c97a785e45701f56110d7c25fa54dcd31a477142d not found: ID does not exist"
Jul 29 21:14:29.212744 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.212720    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"]
Jul 29 21:14:29.216809 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.216788    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6947d68e-kserve-58c8c45b87-jsqz8"]
Jul 29 21:14:29.255493 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.255469    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b189937e-9131-4261-8ebb-bccfa463773e" path="/var/lib/kubelet/pods/b189937e-9131-4261-8ebb-bccfa463773e/volumes"
Jul 29 21:14:29.986704 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:29.986648    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw" podUID="00f3039e-8bef-42e0-8ec2-90941c9ec9e6" containerName="main" containerID="cri-o://ee9d25cefb8dee3e04cbb19b8fe801a926eb485bdab2dabd5be6d5e09703551c" gracePeriod=45
Jul 29 21:14:30.201230 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.201196    2579 generic.go:356] "Generic (PLEG): container finished" podID="00f3039e-8bef-42e0-8ec2-90941c9ec9e6" containerID="ee9d25cefb8dee3e04cbb19b8fe801a926eb485bdab2dabd5be6d5e09703551c" exitCode=0
Jul 29 21:14:30.201392 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.201262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw" event={"ID":"00f3039e-8bef-42e0-8ec2-90941c9ec9e6","Type":"ContainerDied","Data":"ee9d25cefb8dee3e04cbb19b8fe801a926eb485bdab2dabd5be6d5e09703551c"}
Jul 29 21:14:30.233242 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.233226    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:14:30.289510 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289454    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tmp-dir\") pod \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") "
Jul 29 21:14:30.289510 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289481    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-home\") pod \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") "
Jul 29 21:14:30.289703 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289526    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mn5ng\" (UniqueName: \"kubernetes.io/projected/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-kube-api-access-mn5ng\") pod \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") "
Jul 29 21:14:30.289703 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289546    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-model-cache\") pod \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") "
Jul 29 21:14:30.289703 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289560    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tls-certs\") pod \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") "
Jul 29 21:14:30.289703 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289587    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-dshm\") pod \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\" (UID: \"00f3039e-8bef-42e0-8ec2-90941c9ec9e6\") "
Jul 29 21:14:30.289898 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289743    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00f3039e-8bef-42e0-8ec2-90941c9ec9e6" (UID: "00f3039e-8bef-42e0-8ec2-90941c9ec9e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:30.289898 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289757    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "00f3039e-8bef-42e0-8ec2-90941c9ec9e6" (UID: "00f3039e-8bef-42e0-8ec2-90941c9ec9e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:30.289898 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289772    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-home" (OuterVolumeSpecName: "home") pod "00f3039e-8bef-42e0-8ec2-90941c9ec9e6" (UID: "00f3039e-8bef-42e0-8ec2-90941c9ec9e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:30.289898 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289873    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:30.289898 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289891    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:30.290055 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.289903    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:30.291554 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.291529    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00f3039e-8bef-42e0-8ec2-90941c9ec9e6" (UID: "00f3039e-8bef-42e0-8ec2-90941c9ec9e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:14:30.291613 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.291589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-kube-api-access-mn5ng" (OuterVolumeSpecName: "kube-api-access-mn5ng") pod "00f3039e-8bef-42e0-8ec2-90941c9ec9e6" (UID: "00f3039e-8bef-42e0-8ec2-90941c9ec9e6"). InnerVolumeSpecName "kube-api-access-mn5ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:14:30.291648 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.291620    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-dshm" (OuterVolumeSpecName: "dshm") pod "00f3039e-8bef-42e0-8ec2-90941c9ec9e6" (UID: "00f3039e-8bef-42e0-8ec2-90941c9ec9e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:14:30.391194 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.391163    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn5ng\" (UniqueName: \"kubernetes.io/projected/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-kube-api-access-mn5ng\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:30.391194 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.391193    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:30.391194 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:30.391203    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f3039e-8bef-42e0-8ec2-90941c9ec9e6-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:14:31.207695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:31.207667    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"
Jul 29 21:14:31.207695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:31.207678    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw" event={"ID":"00f3039e-8bef-42e0-8ec2-90941c9ec9e6","Type":"ContainerDied","Data":"15d7306b3fb3ed27aa7045772562162e1a0bc92351e9f4bcfda227aabb97545a"}
Jul 29 21:14:31.208137 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:31.207726    2579 scope.go:117] "RemoveContainer" containerID="ee9d25cefb8dee3e04cbb19b8fe801a926eb485bdab2dabd5be6d5e09703551c"
Jul 29 21:14:31.231418 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:31.231388    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"]
Jul 29 21:14:31.235588 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:31.235569    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-6947d68e-kserve-69fc88b679-x6lhw"]
Jul 29 21:14:31.255944 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:14:31.255924    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00f3039e-8bef-42e0-8ec2-90941c9ec9e6" path="/var/lib/kubelet/pods/00f3039e-8bef-42e0-8ec2-90941c9ec9e6/volumes"
Jul 29 21:24:11.354703 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.354673    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"]
Jul 29 21:24:11.355343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.354972    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b189937e-9131-4261-8ebb-bccfa463773e" containerName="main"
Jul 29 21:24:11.355343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.354982    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b189937e-9131-4261-8ebb-bccfa463773e" containerName="main"
Jul 29 21:24:11.355343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.354990    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00f3039e-8bef-42e0-8ec2-90941c9ec9e6" containerName="main"
Jul 29 21:24:11.355343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.354996    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00f3039e-8bef-42e0-8ec2-90941c9ec9e6" containerName="main"
Jul 29 21:24:11.355343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.355061    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b189937e-9131-4261-8ebb-bccfa463773e" containerName="main"
Jul 29 21:24:11.355343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.355067    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00f3039e-8bef-42e0-8ec2-90941c9ec9e6" containerName="main"
Jul 29 21:24:11.357867 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.357853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.360132 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.360109    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ac04a8f1\"/\"kube-root-ca.crt\""
Jul 29 21:24:11.361034 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.361015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ac04a8f1\"/\"default-dockercfg-jpwhw\""
Jul 29 21:24:11.361138 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.361033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ac04a8f1\"/\"e2e-lifecycle-model-ac04a8f1-kserve-self-signed-certs\""
Jul 29 21:24:11.361138 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.361023    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ac04a8f1\"/\"openshift-service-ca.crt\""
Jul 29 21:24:11.368055 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.368034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"]
Jul 29 21:24:11.412377 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.412357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49e48e1-8976-4d24-a49a-faead73ed4c9-tls-certs\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.412487 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.412392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-model-cache\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.412487 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.412410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-home\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.412487 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.412424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-dshm\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.412609 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.412528    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-tmp-dir\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.412609 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.412556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt2cn\" (UniqueName: \"kubernetes.io/projected/c49e48e1-8976-4d24-a49a-faead73ed4c9-kube-api-access-wt2cn\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.513839 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.513812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-tmp-dir\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.513954 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.513844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt2cn\" (UniqueName: \"kubernetes.io/projected/c49e48e1-8976-4d24-a49a-faead73ed4c9-kube-api-access-wt2cn\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.513954 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.513870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49e48e1-8976-4d24-a49a-faead73ed4c9-tls-certs\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.513954 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.513893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-model-cache\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.513954 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.513909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-home\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.513954 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.513922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-dshm\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.514196 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.514177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-tmp-dir\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.514325 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.514282    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-model-cache\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.514392 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.514339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-home\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.515980 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.515962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-dshm\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.516212 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.516196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49e48e1-8976-4d24-a49a-faead73ed4c9-tls-certs\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.521682 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.521661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt2cn\" (UniqueName: \"kubernetes.io/projected/c49e48e1-8976-4d24-a49a-faead73ed4c9-kube-api-access-wt2cn\") pod \"e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") " pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.667398 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.667337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:11.799267 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.799244    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"]
Jul 29 21:24:11.801009 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:24:11.800982    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc49e48e1_8976_4d24_a49a_faead73ed4c9.slice/crio-af9a376ab1843afb9da0fa7794cfc89eae3c4ce1b97c701612ca0122ecc86e92 WatchSource:0}: Error finding container af9a376ab1843afb9da0fa7794cfc89eae3c4ce1b97c701612ca0122ecc86e92: Status 404 returned error can't find the container with id af9a376ab1843afb9da0fa7794cfc89eae3c4ce1b97c701612ca0122ecc86e92
Jul 29 21:24:11.863580 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.863554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l" event={"ID":"c49e48e1-8976-4d24-a49a-faead73ed4c9","Type":"ContainerStarted","Data":"c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629"}
Jul 29 21:24:11.863693 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.863592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l" event={"ID":"c49e48e1-8976-4d24-a49a-faead73ed4c9","Type":"ContainerStarted","Data":"af9a376ab1843afb9da0fa7794cfc89eae3c4ce1b97c701612ca0122ecc86e92"}
Jul 29 21:24:11.882989 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:11.882935    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l" podStartSLOduration=0.882917442 podStartE2EDuration="882.917442ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:24:11.881433054 +0000 UTC m=+2201.197921970" watchObservedRunningTime="2026-07-29 21:24:11.882917442 +0000 UTC m=+2201.199406357"
Jul 29 21:24:21.667890 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:21.667859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:21.668497 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:21.668100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:21.680074 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:21.680051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:21.919031 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:21.918961    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:24:59.377907 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:24:59.377826    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"]
Jul 29 21:25:14.396386 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.396350    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l" podUID="c49e48e1-8976-4d24-a49a-faead73ed4c9" containerName="main" containerID="cri-o://c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629" gracePeriod=45
Jul 29 21:25:14.638615 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.638591    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:25:14.723572 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-home\") pod \"c49e48e1-8976-4d24-a49a-faead73ed4c9\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") "
Jul 29 21:25:14.723572 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-tmp-dir\") pod \"c49e48e1-8976-4d24-a49a-faead73ed4c9\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") "
Jul 29 21:25:14.723572 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723568    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49e48e1-8976-4d24-a49a-faead73ed4c9-tls-certs\") pod \"c49e48e1-8976-4d24-a49a-faead73ed4c9\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") "
Jul 29 21:25:14.723824 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-model-cache\") pod \"c49e48e1-8976-4d24-a49a-faead73ed4c9\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") "
Jul 29 21:25:14.723824 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723619    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt2cn\" (UniqueName: \"kubernetes.io/projected/c49e48e1-8976-4d24-a49a-faead73ed4c9-kube-api-access-wt2cn\") pod \"c49e48e1-8976-4d24-a49a-faead73ed4c9\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") "
Jul 29 21:25:14.723824 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-dshm\") pod \"c49e48e1-8976-4d24-a49a-faead73ed4c9\" (UID: \"c49e48e1-8976-4d24-a49a-faead73ed4c9\") "
Jul 29 21:25:14.723967 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c49e48e1-8976-4d24-a49a-faead73ed4c9" (UID: "c49e48e1-8976-4d24-a49a-faead73ed4c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:25:14.723967 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-home" (OuterVolumeSpecName: "home") pod "c49e48e1-8976-4d24-a49a-faead73ed4c9" (UID: "c49e48e1-8976-4d24-a49a-faead73ed4c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:25:14.723967 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c49e48e1-8976-4d24-a49a-faead73ed4c9" (UID: "c49e48e1-8976-4d24-a49a-faead73ed4c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:25:14.724079 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723966    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:25:14.724079 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.723984    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:25:14.724079 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.724001    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:25:14.725786 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.725756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49e48e1-8976-4d24-a49a-faead73ed4c9-kube-api-access-wt2cn" (OuterVolumeSpecName: "kube-api-access-wt2cn") pod "c49e48e1-8976-4d24-a49a-faead73ed4c9" (UID: "c49e48e1-8976-4d24-a49a-faead73ed4c9"). InnerVolumeSpecName "kube-api-access-wt2cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:25:14.725885 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.725784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-dshm" (OuterVolumeSpecName: "dshm") pod "c49e48e1-8976-4d24-a49a-faead73ed4c9" (UID: "c49e48e1-8976-4d24-a49a-faead73ed4c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:25:14.725885 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.725784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c49e48e1-8976-4d24-a49a-faead73ed4c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c49e48e1-8976-4d24-a49a-faead73ed4c9" (UID: "c49e48e1-8976-4d24-a49a-faead73ed4c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:25:14.824911 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.824884    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c49e48e1-8976-4d24-a49a-faead73ed4c9-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:25:14.824911 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.824908    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt2cn\" (UniqueName: \"kubernetes.io/projected/c49e48e1-8976-4d24-a49a-faead73ed4c9-kube-api-access-wt2cn\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:25:14.825082 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:14.824921    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c49e48e1-8976-4d24-a49a-faead73ed4c9-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:25:15.163524 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.163496    2579 generic.go:356] "Generic (PLEG): container finished" podID="c49e48e1-8976-4d24-a49a-faead73ed4c9" containerID="c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629" exitCode=0
Jul 29 21:25:15.163648 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.163536    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"
Jul 29 21:25:15.163648 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.163560    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l" event={"ID":"c49e48e1-8976-4d24-a49a-faead73ed4c9","Type":"ContainerDied","Data":"c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629"}
Jul 29 21:25:15.163648 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.163591    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l" event={"ID":"c49e48e1-8976-4d24-a49a-faead73ed4c9","Type":"ContainerDied","Data":"af9a376ab1843afb9da0fa7794cfc89eae3c4ce1b97c701612ca0122ecc86e92"}
Jul 29 21:25:15.163648 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.163608    2579 scope.go:117] "RemoveContainer" containerID="c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629"
Jul 29 21:25:15.172748 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.172733    2579 scope.go:117] "RemoveContainer" containerID="c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629"
Jul 29 21:25:15.173038 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:25:15.173019    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629\": container with ID starting with c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629 not found: ID does not exist" containerID="c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629"
Jul 29 21:25:15.173087 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.173046    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629"} err="failed to get container status \"c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629\": rpc error: code = NotFound desc = could not find container \"c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629\": container with ID starting with c36bb7943edb1f84c3d2e3416b1f7b009fe8016fd727131cf758446a66495629 not found: ID does not exist"
Jul 29 21:25:15.189691 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.189668    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"]
Jul 29 21:25:15.194960 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.194942    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ac04a8f1/e2e-lifecycle-model-ac04a8f1-kserve-59f8444b99-54z4l"]
Jul 29 21:25:15.255763 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:25:15.255743    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c49e48e1-8976-4d24-a49a-faead73ed4c9" path="/var/lib/kubelet/pods/c49e48e1-8976-4d24-a49a-faead73ed4c9/volumes"
Jul 29 21:27:50.375833 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.375801    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"]
Jul 29 21:27:50.376243 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.376128    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c49e48e1-8976-4d24-a49a-faead73ed4c9" containerName="main"
Jul 29 21:27:50.376243 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.376139    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c49e48e1-8976-4d24-a49a-faead73ed4c9" containerName="main"
Jul 29 21:27:50.376243 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.376199    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c49e48e1-8976-4d24-a49a-faead73ed4c9" containerName="main"
Jul 29 21:27:50.379079 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.379064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.382500 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.382477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d12bd\"/\"default-dockercfg-2qng7\""
Jul 29 21:27:50.382643 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.382505    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d12bd\"/\"openshift-service-ca.crt\""
Jul 29 21:27:50.382643 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.382534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4d12bd\"/\"test-model-4d12bd-kserve-self-signed-certs\""
Jul 29 21:27:50.382643 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.382596    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4d12bd\"/\"kube-root-ca.crt\""
Jul 29 21:27:50.390830 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.390812    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"]
Jul 29 21:27:50.518821 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.518796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-home\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.519016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.518829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5708b490-b3d5-487f-875e-112ee86afa7a-tls-certs\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.519016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.518863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-dshm\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.519016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.518908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-tmp-dir\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.519016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.518966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-model-cache\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.519202 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.519023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qn2\" (UniqueName: \"kubernetes.io/projected/5708b490-b3d5-487f-875e-112ee86afa7a-kube-api-access-k2qn2\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.619450 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619420    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-home\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.619606 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619457    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5708b490-b3d5-487f-875e-112ee86afa7a-tls-certs\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.619606 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-dshm\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.619606 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-tmp-dir\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.619766 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-model-cache\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.619766 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qn2\" (UniqueName: \"kubernetes.io/projected/5708b490-b3d5-487f-875e-112ee86afa7a-kube-api-access-k2qn2\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.619921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-home\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.620041 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-tmp-dir\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.620041 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.619979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-model-cache\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.621701 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.621677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-dshm\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.621886 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.621870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5708b490-b3d5-487f-875e-112ee86afa7a-tls-certs\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.628970 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.628910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qn2\" (UniqueName: \"kubernetes.io/projected/5708b490-b3d5-487f-875e-112ee86afa7a-kube-api-access-k2qn2\") pod \"test-model-4d12bd-kserve-f4f54c68c-xh57h\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") " pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.687934 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.687906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:27:50.818886 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.818703    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"]
Jul 29 21:27:50.821702 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:27:50.821674    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5708b490_b3d5_487f_875e_112ee86afa7a.slice/crio-bbfd14305152f9f6bb062a154c1cc50eff3fba40b0e698e90b274d5b65c58362 WatchSource:0}: Error finding container bbfd14305152f9f6bb062a154c1cc50eff3fba40b0e698e90b274d5b65c58362: Status 404 returned error can't find the container with id bbfd14305152f9f6bb062a154c1cc50eff3fba40b0e698e90b274d5b65c58362
Jul 29 21:27:50.872840 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:50.872809    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h" event={"ID":"5708b490-b3d5-487f-875e-112ee86afa7a","Type":"ContainerStarted","Data":"bbfd14305152f9f6bb062a154c1cc50eff3fba40b0e698e90b274d5b65c58362"}
Jul 29 21:27:51.878969 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:51.878937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h" event={"ID":"5708b490-b3d5-487f-875e-112ee86afa7a","Type":"ContainerStarted","Data":"23a89cda86d12a9f039ea07159d1d71dec93bd8bc22b5d9b58228e7cde72e6c3"}
Jul 29 21:27:51.906056 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:27:51.906005    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h" podStartSLOduration=1.905991872 podStartE2EDuration="1.905991872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:51.905285113 +0000 UTC m=+2421.221774027" watchObservedRunningTime="2026-07-29 21:27:51.905991872 +0000 UTC m=+2421.222480789"
Jul 29 21:28:00.688495 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:00.688417    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:28:00.688495 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:00.688458    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:28:00.700850 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:00.700823    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:28:00.932458 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:00.932428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:28:08.962607 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:08.962574    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"]
Jul 29 21:28:08.965648 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:08.965634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:08.968337 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:08.968316    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-60a3ea\"/\"kube-root-ca.crt\""
Jul 29 21:28:08.969257 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:08.969233    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-60a3ea\"/\"default-dockercfg-n8jvs\""
Jul 29 21:28:08.969355 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:08.969281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-60a3ea\"/\"test-model-60a3ea-kserve-self-signed-certs\""
Jul 29 21:28:08.969355 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:08.969333    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-60a3ea\"/\"openshift-service-ca.crt\""
Jul 29 21:28:08.981671 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:08.981650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"]
Jul 29 21:28:09.066829 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.066806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e91b2a-47dd-4827-ab63-a61343f61dc7-tls-certs\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.066943 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.066847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmln\" (UniqueName: \"kubernetes.io/projected/74e91b2a-47dd-4827-ab63-a61343f61dc7-kube-api-access-5hmln\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.066943 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.066890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-model-cache\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.066943 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.066911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-home\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.066943 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.066932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-dshm\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.067088 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.066951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-tmp-dir\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168069 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e91b2a-47dd-4827-ab63-a61343f61dc7-tls-certs\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168187 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168083    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmln\" (UniqueName: \"kubernetes.io/projected/74e91b2a-47dd-4827-ab63-a61343f61dc7-kube-api-access-5hmln\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168187 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-model-cache\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168187 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-home\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168187 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-dshm\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168432 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-tmp-dir\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168612 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-model-cache\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168673 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168593    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-home\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.168673 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.168630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-tmp-dir\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.170344 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.170322    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-dshm\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.170601 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.170585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e91b2a-47dd-4827-ab63-a61343f61dc7-tls-certs\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.177632 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.177610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmln\" (UniqueName: \"kubernetes.io/projected/74e91b2a-47dd-4827-ab63-a61343f61dc7-kube-api-access-5hmln\") pod \"test-model-60a3ea-kserve-65b95b9d49-42dx9\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") " pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.278469 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.278411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:09.419774 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.419651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"]
Jul 29 21:28:09.421966 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:28:09.421941    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74e91b2a_47dd_4827_ab63_a61343f61dc7.slice/crio-3742090c2c19c19bfb0759c3acdd32fd803c65eecc9906be954699d4192034cf WatchSource:0}: Error finding container 3742090c2c19c19bfb0759c3acdd32fd803c65eecc9906be954699d4192034cf: Status 404 returned error can't find the container with id 3742090c2c19c19bfb0759c3acdd32fd803c65eecc9906be954699d4192034cf
Jul 29 21:28:09.967128 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.967099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9" event={"ID":"74e91b2a-47dd-4827-ab63-a61343f61dc7","Type":"ContainerStarted","Data":"475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51"}
Jul 29 21:28:09.967543 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.967137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9" event={"ID":"74e91b2a-47dd-4827-ab63-a61343f61dc7","Type":"ContainerStarted","Data":"3742090c2c19c19bfb0759c3acdd32fd803c65eecc9906be954699d4192034cf"}
Jul 29 21:28:09.986304 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:09.986251    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9" podStartSLOduration=1.98623733 podStartE2EDuration="1.98623733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:28:09.98502444 +0000 UTC m=+2439.301513356" watchObservedRunningTime="2026-07-29 21:28:09.98623733 +0000 UTC m=+2439.302726244"
Jul 29 21:28:19.278975 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:19.278943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:19.279405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:19.278987    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:19.291664 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:19.291641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:20.024158 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:20.024127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:28:58.017457 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:58.017426    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"]
Jul 29 21:28:58.964462 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:28:58.964434    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"]
Jul 29 21:29:13.036049 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.035986    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h" podUID="5708b490-b3d5-487f-875e-112ee86afa7a" containerName="main" containerID="cri-o://23a89cda86d12a9f039ea07159d1d71dec93bd8bc22b5d9b58228e7cde72e6c3" gracePeriod=45
Jul 29 21:29:13.263531 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.263504    2579 generic.go:356] "Generic (PLEG): container finished" podID="5708b490-b3d5-487f-875e-112ee86afa7a" containerID="23a89cda86d12a9f039ea07159d1d71dec93bd8bc22b5d9b58228e7cde72e6c3" exitCode=0
Jul 29 21:29:13.263655 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.263569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h" event={"ID":"5708b490-b3d5-487f-875e-112ee86afa7a","Type":"ContainerDied","Data":"23a89cda86d12a9f039ea07159d1d71dec93bd8bc22b5d9b58228e7cde72e6c3"}
Jul 29 21:29:13.279485 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.279467    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:29:13.363492 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363464    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2qn2\" (UniqueName: \"kubernetes.io/projected/5708b490-b3d5-487f-875e-112ee86afa7a-kube-api-access-k2qn2\") pod \"5708b490-b3d5-487f-875e-112ee86afa7a\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") "
Jul 29 21:29:13.363639 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-dshm\") pod \"5708b490-b3d5-487f-875e-112ee86afa7a\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") "
Jul 29 21:29:13.363639 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5708b490-b3d5-487f-875e-112ee86afa7a-tls-certs\") pod \"5708b490-b3d5-487f-875e-112ee86afa7a\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") "
Jul 29 21:29:13.363768 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-home\") pod \"5708b490-b3d5-487f-875e-112ee86afa7a\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") "
Jul 29 21:29:13.363768 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363713    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-tmp-dir\") pod \"5708b490-b3d5-487f-875e-112ee86afa7a\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") "
Jul 29 21:29:13.363768 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-model-cache\") pod \"5708b490-b3d5-487f-875e-112ee86afa7a\" (UID: \"5708b490-b3d5-487f-875e-112ee86afa7a\") "
Jul 29 21:29:13.363933 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-home" (OuterVolumeSpecName: "home") pod "5708b490-b3d5-487f-875e-112ee86afa7a" (UID: "5708b490-b3d5-487f-875e-112ee86afa7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:13.363990 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.363970    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5708b490-b3d5-487f-875e-112ee86afa7a" (UID: "5708b490-b3d5-487f-875e-112ee86afa7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:13.364085 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.364061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5708b490-b3d5-487f-875e-112ee86afa7a" (UID: "5708b490-b3d5-487f-875e-112ee86afa7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:13.364085 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.364075    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:13.364199 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.364114    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:13.365706 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.365677    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5708b490-b3d5-487f-875e-112ee86afa7a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5708b490-b3d5-487f-875e-112ee86afa7a" (UID: "5708b490-b3d5-487f-875e-112ee86afa7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:29:13.365706 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.365690    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-dshm" (OuterVolumeSpecName: "dshm") pod "5708b490-b3d5-487f-875e-112ee86afa7a" (UID: "5708b490-b3d5-487f-875e-112ee86afa7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:13.365862 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.365751    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5708b490-b3d5-487f-875e-112ee86afa7a-kube-api-access-k2qn2" (OuterVolumeSpecName: "kube-api-access-k2qn2") pod "5708b490-b3d5-487f-875e-112ee86afa7a" (UID: "5708b490-b3d5-487f-875e-112ee86afa7a"). InnerVolumeSpecName "kube-api-access-k2qn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:29:13.465111 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.465089    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:13.465111 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.465108    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2qn2\" (UniqueName: \"kubernetes.io/projected/5708b490-b3d5-487f-875e-112ee86afa7a-kube-api-access-k2qn2\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:13.465233 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.465118    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5708b490-b3d5-487f-875e-112ee86afa7a-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:13.465233 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.465126    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5708b490-b3d5-487f-875e-112ee86afa7a-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:13.982922 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:13.982882    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9" podUID="74e91b2a-47dd-4827-ab63-a61343f61dc7" containerName="main" containerID="cri-o://475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51" gracePeriod=45
Jul 29 21:29:14.245071 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.245014    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:29:14.270080 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.270052    2579 generic.go:356] "Generic (PLEG): container finished" podID="74e91b2a-47dd-4827-ab63-a61343f61dc7" containerID="475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51" exitCode=0
Jul 29 21:29:14.270472 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.270099    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"
Jul 29 21:29:14.270472 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.270129    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9" event={"ID":"74e91b2a-47dd-4827-ab63-a61343f61dc7","Type":"ContainerDied","Data":"475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51"}
Jul 29 21:29:14.270472 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.270167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9" event={"ID":"74e91b2a-47dd-4827-ab63-a61343f61dc7","Type":"ContainerDied","Data":"3742090c2c19c19bfb0759c3acdd32fd803c65eecc9906be954699d4192034cf"}
Jul 29 21:29:14.270472 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.270188    2579 scope.go:117] "RemoveContainer" containerID="475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51"
Jul 29 21:29:14.271665 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.271648    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"
Jul 29 21:29:14.271745 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.271693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h" event={"ID":"5708b490-b3d5-487f-875e-112ee86afa7a","Type":"ContainerDied","Data":"bbfd14305152f9f6bb062a154c1cc50eff3fba40b0e698e90b274d5b65c58362"}
Jul 29 21:29:14.281001 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.280984    2579 scope.go:117] "RemoveContainer" containerID="475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51"
Jul 29 21:29:14.281325 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:29:14.281277    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51\": container with ID starting with 475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51 not found: ID does not exist" containerID="475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51"
Jul 29 21:29:14.281483 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.281334    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51"} err="failed to get container status \"475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51\": rpc error: code = NotFound desc = could not find container \"475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51\": container with ID starting with 475d5b232dad7c3d53cf9109588f58cf7785ab89c82383ea2aa47b27f2bf8d51 not found: ID does not exist"
Jul 29 21:29:14.281483 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.281357    2579 scope.go:117] "RemoveContainer" containerID="23a89cda86d12a9f039ea07159d1d71dec93bd8bc22b5d9b58228e7cde72e6c3"
Jul 29 21:29:14.293388 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.293367    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"]
Jul 29 21:29:14.297325 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.297306    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4d12bd/test-model-4d12bd-kserve-f4f54c68c-xh57h"]
Jul 29 21:29:14.370871 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.370844    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-model-cache\") pod \"74e91b2a-47dd-4827-ab63-a61343f61dc7\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") "
Jul 29 21:29:14.370994 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.370883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-dshm\") pod \"74e91b2a-47dd-4827-ab63-a61343f61dc7\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") "
Jul 29 21:29:14.370994 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.370941    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-home\") pod \"74e91b2a-47dd-4827-ab63-a61343f61dc7\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") "
Jul 29 21:29:14.370994 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.370965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hmln\" (UniqueName: \"kubernetes.io/projected/74e91b2a-47dd-4827-ab63-a61343f61dc7-kube-api-access-5hmln\") pod \"74e91b2a-47dd-4827-ab63-a61343f61dc7\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") "
Jul 29 21:29:14.371137 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.370998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e91b2a-47dd-4827-ab63-a61343f61dc7-tls-certs\") pod \"74e91b2a-47dd-4827-ab63-a61343f61dc7\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") "
Jul 29 21:29:14.371137 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.371020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-tmp-dir\") pod \"74e91b2a-47dd-4827-ab63-a61343f61dc7\" (UID: \"74e91b2a-47dd-4827-ab63-a61343f61dc7\") "
Jul 29 21:29:14.371250 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.371170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "74e91b2a-47dd-4827-ab63-a61343f61dc7" (UID: "74e91b2a-47dd-4827-ab63-a61343f61dc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:14.371250 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.371220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-home" (OuterVolumeSpecName: "home") pod "74e91b2a-47dd-4827-ab63-a61343f61dc7" (UID: "74e91b2a-47dd-4827-ab63-a61343f61dc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:14.371408 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.371327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74e91b2a-47dd-4827-ab63-a61343f61dc7" (UID: "74e91b2a-47dd-4827-ab63-a61343f61dc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:14.371408 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.371376    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:14.371408 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.371395    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:14.373170 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.373150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e91b2a-47dd-4827-ab63-a61343f61dc7-kube-api-access-5hmln" (OuterVolumeSpecName: "kube-api-access-5hmln") pod "74e91b2a-47dd-4827-ab63-a61343f61dc7" (UID: "74e91b2a-47dd-4827-ab63-a61343f61dc7"). InnerVolumeSpecName "kube-api-access-5hmln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:29:14.373516 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.373497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-dshm" (OuterVolumeSpecName: "dshm") pod "74e91b2a-47dd-4827-ab63-a61343f61dc7" (UID: "74e91b2a-47dd-4827-ab63-a61343f61dc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:29:14.373606 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.373546    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e91b2a-47dd-4827-ab63-a61343f61dc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74e91b2a-47dd-4827-ab63-a61343f61dc7" (UID: "74e91b2a-47dd-4827-ab63-a61343f61dc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:29:14.472069 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.472045    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:14.472069 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.472067    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmln\" (UniqueName: \"kubernetes.io/projected/74e91b2a-47dd-4827-ab63-a61343f61dc7-kube-api-access-5hmln\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:14.472214 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.472078    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74e91b2a-47dd-4827-ab63-a61343f61dc7-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:14.472214 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.472086    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74e91b2a-47dd-4827-ab63-a61343f61dc7-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:14.596094 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.596068    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"]
Jul 29 21:29:14.600406 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:14.600384    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-60a3ea/test-model-60a3ea-kserve-65b95b9d49-42dx9"]
Jul 29 21:29:15.255955 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:15.255924    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5708b490-b3d5-487f-875e-112ee86afa7a" path="/var/lib/kubelet/pods/5708b490-b3d5-487f-875e-112ee86afa7a/volumes"
Jul 29 21:29:15.256342 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:15.256303    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e91b2a-47dd-4827-ab63-a61343f61dc7" path="/var/lib/kubelet/pods/74e91b2a-47dd-4827-ab63-a61343f61dc7/volumes"
Jul 29 21:29:16.615956 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.615928    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"]
Jul 29 21:29:16.616366 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.616317    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e91b2a-47dd-4827-ab63-a61343f61dc7" containerName="main"
Jul 29 21:29:16.616366 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.616335    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e91b2a-47dd-4827-ab63-a61343f61dc7" containerName="main"
Jul 29 21:29:16.616366 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.616355    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5708b490-b3d5-487f-875e-112ee86afa7a" containerName="main"
Jul 29 21:29:16.616366 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.616360    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5708b490-b3d5-487f-875e-112ee86afa7a" containerName="main"
Jul 29 21:29:16.616502 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.616428    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5708b490-b3d5-487f-875e-112ee86afa7a" containerName="main"
Jul 29 21:29:16.616502 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.616436    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e91b2a-47dd-4827-ab63-a61343f61dc7" containerName="main"
Jul 29 21:29:16.620764 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.620748    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"
Jul 29 21:29:16.623396 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.623379    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:29:16.623468 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.623408    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:29:16.629273 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.629251    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"]
Jul 29 21:29:16.788199 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.788171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3eeb59-96f6-4170-a1e4-13813980ec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r\" (UID: \"5d3eeb59-96f6-4170-a1e4-13813980ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"
Jul 29 21:29:16.889250 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.889175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3eeb59-96f6-4170-a1e4-13813980ec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r\" (UID: \"5d3eeb59-96f6-4170-a1e4-13813980ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"
Jul 29 21:29:16.889781 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.889763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3eeb59-96f6-4170-a1e4-13813980ec77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r\" (UID: \"5d3eeb59-96f6-4170-a1e4-13813980ec77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"
Jul 29 21:29:16.931304 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:16.931269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"
Jul 29 21:29:17.063197 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:17.063174    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"]
Jul 29 21:29:17.065042 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:29:17.065017    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3eeb59_96f6_4170_a1e4_13813980ec77.slice/crio-2f210bac6608216b9cc803ed2a2eba45764e6e5ca744a26413771f4d9f083428 WatchSource:0}: Error finding container 2f210bac6608216b9cc803ed2a2eba45764e6e5ca744a26413771f4d9f083428: Status 404 returned error can't find the container with id 2f210bac6608216b9cc803ed2a2eba45764e6e5ca744a26413771f4d9f083428
Jul 29 21:29:17.067163 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:17.067144    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:29:17.286681 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:17.286596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r" event={"ID":"5d3eeb59-96f6-4170-a1e4-13813980ec77","Type":"ContainerStarted","Data":"2f210bac6608216b9cc803ed2a2eba45764e6e5ca744a26413771f4d9f083428"}
Jul 29 21:29:20.306515 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:20.306484    2579 generic.go:356] "Generic (PLEG): container finished" podID="5d3eeb59-96f6-4170-a1e4-13813980ec77" containerID="c165f31022d682670b220b5cdf724a3f75dba1da5dfe578a84cc162308f159fb" exitCode=0
Jul 29 21:29:20.306869 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:20.306568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r" event={"ID":"5d3eeb59-96f6-4170-a1e4-13813980ec77","Type":"ContainerDied","Data":"c165f31022d682670b220b5cdf724a3f75dba1da5dfe578a84cc162308f159fb"}
Jul 29 21:29:21.435858 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:21.435835    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"
Jul 29 21:29:21.530121 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:21.530097    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3eeb59-96f6-4170-a1e4-13813980ec77-maas-api-service-ca\") pod \"5d3eeb59-96f6-4170-a1e4-13813980ec77\" (UID: \"5d3eeb59-96f6-4170-a1e4-13813980ec77\") "
Jul 29 21:29:21.530449 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:21.530429    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3eeb59-96f6-4170-a1e4-13813980ec77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d3eeb59-96f6-4170-a1e4-13813980ec77" (UID: "5d3eeb59-96f6-4170-a1e4-13813980ec77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:29:21.630777 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:21.630753    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3eeb59-96f6-4170-a1e4-13813980ec77-maas-api-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:29:22.317178 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:22.317152    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"
Jul 29 21:29:22.317355 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:22.317203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r" event={"ID":"5d3eeb59-96f6-4170-a1e4-13813980ec77","Type":"ContainerDied","Data":"2f210bac6608216b9cc803ed2a2eba45764e6e5ca744a26413771f4d9f083428"}
Jul 29 21:29:22.317355 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:22.317245    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f210bac6608216b9cc803ed2a2eba45764e6e5ca744a26413771f4d9f083428"
Jul 29 21:29:27.388942 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:27.388863    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"]
Jul 29 21:29:27.395335 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:27.395312    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-60a3ea-6ls7r"]
Jul 29 21:29:29.256545 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:29:29.256505    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d3eeb59-96f6-4170-a1e4-13813980ec77" path="/var/lib/kubelet/pods/5d3eeb59-96f6-4170-a1e4-13813980ec77/volumes"
Jul 29 21:30:35.092123 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.092092    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"]
Jul 29 21:30:35.092539 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.092450    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d3eeb59-96f6-4170-a1e4-13813980ec77" containerName="revoke-keys"
Jul 29 21:30:35.092539 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.092462    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d3eeb59-96f6-4170-a1e4-13813980ec77" containerName="revoke-keys"
Jul 29 21:30:35.092539 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.092522    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d3eeb59-96f6-4170-a1e4-13813980ec77" containerName="revoke-keys"
Jul 29 21:30:35.095033 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.095017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"
Jul 29 21:30:35.097372 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.097346    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:30:35.097372 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.097364    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:30:35.103050 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.103026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"]
Jul 29 21:30:35.120542 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.120519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d2d7a1-51d5-455e-8a92-3fe39b623926-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc\" (UID: \"91d2d7a1-51d5-455e-8a92-3fe39b623926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"
Jul 29 21:30:35.220921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.220897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d2d7a1-51d5-455e-8a92-3fe39b623926-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc\" (UID: \"91d2d7a1-51d5-455e-8a92-3fe39b623926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"
Jul 29 21:30:35.221540 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.221521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d2d7a1-51d5-455e-8a92-3fe39b623926-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc\" (UID: \"91d2d7a1-51d5-455e-8a92-3fe39b623926\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"
Jul 29 21:30:35.405817 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.405753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"
Jul 29 21:30:35.560217 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.560190    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"]
Jul 29 21:30:35.562134 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:30:35.562109    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d2d7a1_51d5_455e_8a92_3fe39b623926.slice/crio-69c175609a6ee8b5f4e09d07836d6836104ff0e16ff7e725c384eb0846d76cd3 WatchSource:0}: Error finding container 69c175609a6ee8b5f4e09d07836d6836104ff0e16ff7e725c384eb0846d76cd3: Status 404 returned error can't find the container with id 69c175609a6ee8b5f4e09d07836d6836104ff0e16ff7e725c384eb0846d76cd3
Jul 29 21:30:35.654558 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:35.654533    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc" event={"ID":"91d2d7a1-51d5-455e-8a92-3fe39b623926","Type":"ContainerStarted","Data":"69c175609a6ee8b5f4e09d07836d6836104ff0e16ff7e725c384eb0846d76cd3"}
Jul 29 21:30:36.661519 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:36.661486    2579 generic.go:356] "Generic (PLEG): container finished" podID="91d2d7a1-51d5-455e-8a92-3fe39b623926" containerID="f651ddfdc2559efa62598eea959d717b6c15a9e079743fb6a7ff509f538010b8" exitCode=0
Jul 29 21:30:36.661941 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:36.661551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc" event={"ID":"91d2d7a1-51d5-455e-8a92-3fe39b623926","Type":"ContainerDied","Data":"f651ddfdc2559efa62598eea959d717b6c15a9e079743fb6a7ff509f538010b8"}
Jul 29 21:30:37.793047 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:37.793020    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"
Jul 29 21:30:37.841714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:37.841690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d2d7a1-51d5-455e-8a92-3fe39b623926-maas-api-service-ca\") pod \"91d2d7a1-51d5-455e-8a92-3fe39b623926\" (UID: \"91d2d7a1-51d5-455e-8a92-3fe39b623926\") "
Jul 29 21:30:37.842034 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:37.842012    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d2d7a1-51d5-455e-8a92-3fe39b623926-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91d2d7a1-51d5-455e-8a92-3fe39b623926" (UID: "91d2d7a1-51d5-455e-8a92-3fe39b623926"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:30:37.942591 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:37.942535    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d2d7a1-51d5-455e-8a92-3fe39b623926-maas-api-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:30:38.672045 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:38.672016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc" event={"ID":"91d2d7a1-51d5-455e-8a92-3fe39b623926","Type":"ContainerDied","Data":"69c175609a6ee8b5f4e09d07836d6836104ff0e16ff7e725c384eb0846d76cd3"}
Jul 29 21:30:38.672045 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:38.672033    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"
Jul 29 21:30:38.672248 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:38.672067    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69c175609a6ee8b5f4e09d07836d6836104ff0e16ff7e725c384eb0846d76cd3"
Jul 29 21:30:50.843039 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:50.843002    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"]
Jul 29 21:30:50.845814 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:50.845791    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7e568-wtdlc"]
Jul 29 21:30:51.256661 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:51.256583    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91d2d7a1-51d5-455e-8a92-3fe39b623926" path="/var/lib/kubelet/pods/91d2d7a1-51d5-455e-8a92-3fe39b623926/volumes"
Jul 29 21:30:57.607735 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.607652    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"]
Jul 29 21:30:57.608311 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.608144    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d2d7a1-51d5-455e-8a92-3fe39b623926" containerName="revoke-keys"
Jul 29 21:30:57.608311 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.608163    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d2d7a1-51d5-455e-8a92-3fe39b623926" containerName="revoke-keys"
Jul 29 21:30:57.608311 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.608277    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d2d7a1-51d5-455e-8a92-3fe39b623926" containerName="revoke-keys"
Jul 29 21:30:57.611388 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.611368    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"
Jul 29 21:30:57.613564 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.613548    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:30:57.613694 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.613676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:30:57.618770 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.618501    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"]
Jul 29 21:30:57.695207 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.695176    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd58d5d1-fb9a-459e-bf55-1e529f234463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc\" (UID: \"dd58d5d1-fb9a-459e-bf55-1e529f234463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"
Jul 29 21:30:57.796154 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.796128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd58d5d1-fb9a-459e-bf55-1e529f234463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc\" (UID: \"dd58d5d1-fb9a-459e-bf55-1e529f234463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"
Jul 29 21:30:57.796686 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.796669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd58d5d1-fb9a-459e-bf55-1e529f234463-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc\" (UID: \"dd58d5d1-fb9a-459e-bf55-1e529f234463\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"
Jul 29 21:30:57.922583 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:57.922502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"
Jul 29 21:30:58.055136 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:58.055108    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"]
Jul 29 21:30:58.056686 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:30:58.056658    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd58d5d1_fb9a_459e_bf55_1e529f234463.slice/crio-ee94efa06d77f15c16467e787ec68964d22c122f95b20c51a3b5831dd541d6e4 WatchSource:0}: Error finding container ee94efa06d77f15c16467e787ec68964d22c122f95b20c51a3b5831dd541d6e4: Status 404 returned error can't find the container with id ee94efa06d77f15c16467e787ec68964d22c122f95b20c51a3b5831dd541d6e4
Jul 29 21:30:58.768081 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:58.768053    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd58d5d1-fb9a-459e-bf55-1e529f234463" containerID="e449d04c69457eef6ff0e9f05b258e76e0450fc7d7dbc52e782244601ca05890" exitCode=0
Jul 29 21:30:58.768445 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:58.768130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc" event={"ID":"dd58d5d1-fb9a-459e-bf55-1e529f234463","Type":"ContainerDied","Data":"e449d04c69457eef6ff0e9f05b258e76e0450fc7d7dbc52e782244601ca05890"}
Jul 29 21:30:58.768445 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:58.768180    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc" event={"ID":"dd58d5d1-fb9a-459e-bf55-1e529f234463","Type":"ContainerStarted","Data":"ee94efa06d77f15c16467e787ec68964d22c122f95b20c51a3b5831dd541d6e4"}
Jul 29 21:30:59.900048 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:30:59.900022    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"
Jul 29 21:31:00.013449 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:00.013420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd58d5d1-fb9a-459e-bf55-1e529f234463-maas-api-service-ca\") pod \"dd58d5d1-fb9a-459e-bf55-1e529f234463\" (UID: \"dd58d5d1-fb9a-459e-bf55-1e529f234463\") "
Jul 29 21:31:00.013787 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:00.013763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd58d5d1-fb9a-459e-bf55-1e529f234463-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd58d5d1-fb9a-459e-bf55-1e529f234463" (UID: "dd58d5d1-fb9a-459e-bf55-1e529f234463"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:31:00.114741 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:00.114715    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd58d5d1-fb9a-459e-bf55-1e529f234463-maas-api-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:31:00.778904 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:00.778873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc" event={"ID":"dd58d5d1-fb9a-459e-bf55-1e529f234463","Type":"ContainerDied","Data":"ee94efa06d77f15c16467e787ec68964d22c122f95b20c51a3b5831dd541d6e4"}
Jul 29 21:31:00.778904 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:00.778891    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"
Jul 29 21:31:00.779103 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:00.778924    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee94efa06d77f15c16467e787ec68964d22c122f95b20c51a3b5831dd541d6e4"
Jul 29 21:31:13.246538 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:13.246500    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"]
Jul 29 21:31:13.249032 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:13.249008    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3b326f-l4rmc"]
Jul 29 21:31:13.255848 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:31:13.255825    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd58d5d1-fb9a-459e-bf55-1e529f234463" path="/var/lib/kubelet/pods/dd58d5d1-fb9a-459e-bf55-1e529f234463/volumes"
Jul 29 21:32:36.853211 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.853182    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"]
Jul 29 21:32:36.853625 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.853499    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd58d5d1-fb9a-459e-bf55-1e529f234463" containerName="revoke-keys"
Jul 29 21:32:36.853625 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.853509    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd58d5d1-fb9a-459e-bf55-1e529f234463" containerName="revoke-keys"
Jul 29 21:32:36.853625 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.853582    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd58d5d1-fb9a-459e-bf55-1e529f234463" containerName="revoke-keys"
Jul 29 21:32:36.856515 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.856500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:36.860013 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.859992    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"auth-test-model-eb3087-kserve-self-signed-certs\""
Jul 29 21:32:36.860136 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.860010    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"default-dockercfg-mg2w8\""
Jul 29 21:32:36.860136 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.860049    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"kube-root-ca.crt\""
Jul 29 21:32:36.860136 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.860050    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"openshift-service-ca.crt\""
Jul 29 21:32:36.865396 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.865374    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"]
Jul 29 21:32:36.970552 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.970526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-dshm\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:36.970695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.970559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-model-cache\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:36.970695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.970587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-tmp-dir\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:36.970695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.970662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2c1d20-c260-402b-a779-8f171421f65c-tls-certs\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:36.970799 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.970725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-home\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:36.970799 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:36.970757    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6px\" (UniqueName: \"kubernetes.io/projected/2d2c1d20-c260-402b-a779-8f171421f65c-kube-api-access-xv6px\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072064 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-home\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072064 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6px\" (UniqueName: \"kubernetes.io/projected/2d2c1d20-c260-402b-a779-8f171421f65c-kube-api-access-xv6px\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072244 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072124    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-dshm\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072244 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-model-cache\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072244 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-tmp-dir\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072383 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2c1d20-c260-402b-a779-8f171421f65c-tls-certs\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072501 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072481    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-home\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072498    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-tmp-dir\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.072626 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.072584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-model-cache\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.074247 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.074226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-dshm\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.074631 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.074612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2c1d20-c260-402b-a779-8f171421f65c-tls-certs\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.081456 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.081435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6px\" (UniqueName: \"kubernetes.io/projected/2d2c1d20-c260-402b-a779-8f171421f65c-kube-api-access-xv6px\") pod \"auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") " pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.165925 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.165862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:37.296036 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:37.295976    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"]
Jul 29 21:32:37.298443 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:32:37.298417    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d2c1d20_c260_402b_a779_8f171421f65c.slice/crio-e1a2889758e25f349fa42d53d80a26b528abc648aaa920e902238397009b1fac WatchSource:0}: Error finding container e1a2889758e25f349fa42d53d80a26b528abc648aaa920e902238397009b1fac: Status 404 returned error can't find the container with id e1a2889758e25f349fa42d53d80a26b528abc648aaa920e902238397009b1fac
Jul 29 21:32:38.239798 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:38.239761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2" event={"ID":"2d2c1d20-c260-402b-a779-8f171421f65c","Type":"ContainerStarted","Data":"7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8"}
Jul 29 21:32:38.240171 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:38.239812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2" event={"ID":"2d2c1d20-c260-402b-a779-8f171421f65c","Type":"ContainerStarted","Data":"e1a2889758e25f349fa42d53d80a26b528abc648aaa920e902238397009b1fac"}
Jul 29 21:32:38.259298 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:38.259243    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2" podStartSLOduration=2.259227814 podStartE2EDuration="2.259227814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:32:38.258453204 +0000 UTC m=+2707.574942129" watchObservedRunningTime="2026-07-29 21:32:38.259227814 +0000 UTC m=+2707.575716729"
Jul 29 21:32:47.166829 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:47.166796    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:47.167342 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:47.166850    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:47.178917 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:47.178896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:32:47.292843 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:32:47.292815    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:33:00.880184 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.880155    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"]
Jul 29 21:33:00.883430 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.883412    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:00.885961 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.885930    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"kube-root-ca.crt\""
Jul 29 21:33:00.886920 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.886894    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"default-dockercfg-pbrzs\""
Jul 29 21:33:00.887016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.886898    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"auth-test-model-46bb38-kserve-self-signed-certs\""
Jul 29 21:33:00.887016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.886952    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"openshift-service-ca.crt\""
Jul 29 21:33:00.897830 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.897808    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"]
Jul 29 21:33:00.963866 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.963837    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-dshm\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:00.963967 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.963872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tmp-dir\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:00.963967 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.963897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tls-certs\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:00.963967 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.963922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx48d\" (UniqueName: \"kubernetes.io/projected/28ef21d7-1e23-4ec2-85bc-87f3708ece31-kube-api-access-vx48d\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:00.964074 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.963964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-home\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:00.964074 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:00.964002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-model-cache\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.064562 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.064537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tls-certs\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.064721 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.064570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx48d\" (UniqueName: \"kubernetes.io/projected/28ef21d7-1e23-4ec2-85bc-87f3708ece31-kube-api-access-vx48d\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.064721 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.064589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-home\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.064721 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.064699    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-model-cache\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.064912 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.064747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-dshm\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.064912 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.064792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tmp-dir\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.065044 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.065027    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-model-cache\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.065124 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.065104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-home\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.065176 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.065162    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tmp-dir\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.066841 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.066822    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-dshm\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.067061 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.067044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tls-certs\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.073719 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.073700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx48d\" (UniqueName: \"kubernetes.io/projected/28ef21d7-1e23-4ec2-85bc-87f3708ece31-kube-api-access-vx48d\") pod \"auth-test-model-46bb38-kserve-b866b757f-8ffrs\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") " pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.193386 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.193319    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:01.326203 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.326180    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"]
Jul 29 21:33:01.327827 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:33:01.327801    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28ef21d7_1e23_4ec2_85bc_87f3708ece31.slice/crio-495be55f90a3268d8cc14c0ae9ce3b1abc0247f09e15af25cf70d4170e87be95 WatchSource:0}: Error finding container 495be55f90a3268d8cc14c0ae9ce3b1abc0247f09e15af25cf70d4170e87be95: Status 404 returned error can't find the container with id 495be55f90a3268d8cc14c0ae9ce3b1abc0247f09e15af25cf70d4170e87be95
Jul 29 21:33:01.348186 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:01.348162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs" event={"ID":"28ef21d7-1e23-4ec2-85bc-87f3708ece31","Type":"ContainerStarted","Data":"495be55f90a3268d8cc14c0ae9ce3b1abc0247f09e15af25cf70d4170e87be95"}
Jul 29 21:33:02.355004 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:02.354965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs" event={"ID":"28ef21d7-1e23-4ec2-85bc-87f3708ece31","Type":"ContainerStarted","Data":"9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9"}
Jul 29 21:33:02.378302 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:02.378242    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs" podStartSLOduration=2.378226385 podStartE2EDuration="2.378226385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:33:02.374504413 +0000 UTC m=+2731.690993327" watchObservedRunningTime="2026-07-29 21:33:02.378226385 +0000 UTC m=+2731.694715300"
Jul 29 21:33:11.193631 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:11.193581    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:11.193631 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:11.193643    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:11.205903 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:11.205880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:33:11.407557 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:33:11.407529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:35:31.437130 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:35:31.437063    2579 scope.go:117] "RemoveContainer" containerID="c165f31022d682670b220b5cdf724a3f75dba1da5dfe578a84cc162308f159fb"
Jul 29 21:37:31.465416 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:31.465389    2579 scope.go:117] "RemoveContainer" containerID="e449d04c69457eef6ff0e9f05b258e76e0450fc7d7dbc52e782244601ca05890"
Jul 29 21:37:31.473376 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:31.473357    2579 scope.go:117] "RemoveContainer" containerID="f651ddfdc2559efa62598eea959d717b6c15a9e079743fb6a7ff509f538010b8"
Jul 29 21:37:35.964349 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:35.964315    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"]
Jul 29 21:37:36.903143 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:36.903099    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"]
Jul 29 21:37:50.983047 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:50.982997    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2" podUID="2d2c1d20-c260-402b-a779-8f171421f65c" containerName="main" containerID="cri-o://7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8" gracePeriod=45
Jul 29 21:37:51.230697 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.230670    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:37:51.312941 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.312862    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-dshm\") pod \"2d2c1d20-c260-402b-a779-8f171421f65c\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") "
Jul 29 21:37:51.312941 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.312893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-model-cache\") pod \"2d2c1d20-c260-402b-a779-8f171421f65c\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") "
Jul 29 21:37:51.312941 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.312930    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-tmp-dir\") pod \"2d2c1d20-c260-402b-a779-8f171421f65c\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") "
Jul 29 21:37:51.313198 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.312958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2c1d20-c260-402b-a779-8f171421f65c-tls-certs\") pod \"2d2c1d20-c260-402b-a779-8f171421f65c\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") "
Jul 29 21:37:51.313198 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313011    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-home\") pod \"2d2c1d20-c260-402b-a779-8f171421f65c\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") "
Jul 29 21:37:51.313198 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313055    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv6px\" (UniqueName: \"kubernetes.io/projected/2d2c1d20-c260-402b-a779-8f171421f65c-kube-api-access-xv6px\") pod \"2d2c1d20-c260-402b-a779-8f171421f65c\" (UID: \"2d2c1d20-c260-402b-a779-8f171421f65c\") "
Jul 29 21:37:51.313389 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d2c1d20-c260-402b-a779-8f171421f65c" (UID: "2d2c1d20-c260-402b-a779-8f171421f65c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:51.313389 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d2c1d20-c260-402b-a779-8f171421f65c" (UID: "2d2c1d20-c260-402b-a779-8f171421f65c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:51.313389 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313367    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-home" (OuterVolumeSpecName: "home") pod "2d2c1d20-c260-402b-a779-8f171421f65c" (UID: "2d2c1d20-c260-402b-a779-8f171421f65c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:51.313533 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313446    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:51.313533 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313465    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:51.313533 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.313482    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:51.315158 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.315134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2c1d20-c260-402b-a779-8f171421f65c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d2c1d20-c260-402b-a779-8f171421f65c" (UID: "2d2c1d20-c260-402b-a779-8f171421f65c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:37:51.315158 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.315137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-dshm" (OuterVolumeSpecName: "dshm") pod "2d2c1d20-c260-402b-a779-8f171421f65c" (UID: "2d2c1d20-c260-402b-a779-8f171421f65c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:51.315280 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.315222    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2c1d20-c260-402b-a779-8f171421f65c-kube-api-access-xv6px" (OuterVolumeSpecName: "kube-api-access-xv6px") pod "2d2c1d20-c260-402b-a779-8f171421f65c" (UID: "2d2c1d20-c260-402b-a779-8f171421f65c"). InnerVolumeSpecName "kube-api-access-xv6px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:37:51.414384 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.414356    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d2c1d20-c260-402b-a779-8f171421f65c-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:51.414384 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.414381    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d2c1d20-c260-402b-a779-8f171421f65c-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:51.414384 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.414393    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6px\" (UniqueName: \"kubernetes.io/projected/2d2c1d20-c260-402b-a779-8f171421f65c-kube-api-access-xv6px\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:51.761280 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.761243    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d2c1d20-c260-402b-a779-8f171421f65c" containerID="7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8" exitCode=0
Jul 29 21:37:51.761489 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.761308    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"
Jul 29 21:37:51.761489 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.761335    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2" event={"ID":"2d2c1d20-c260-402b-a779-8f171421f65c","Type":"ContainerDied","Data":"7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8"}
Jul 29 21:37:51.761489 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.761378    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2" event={"ID":"2d2c1d20-c260-402b-a779-8f171421f65c","Type":"ContainerDied","Data":"e1a2889758e25f349fa42d53d80a26b528abc648aaa920e902238397009b1fac"}
Jul 29 21:37:51.761489 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.761396    2579 scope.go:117] "RemoveContainer" containerID="7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8"
Jul 29 21:37:51.770319 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.770281    2579 scope.go:117] "RemoveContainer" containerID="7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8"
Jul 29 21:37:51.770576 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:37:51.770559    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8\": container with ID starting with 7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8 not found: ID does not exist" containerID="7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8"
Jul 29 21:37:51.770648 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.770584    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8"} err="failed to get container status \"7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8\": rpc error: code = NotFound desc = could not find container \"7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8\": container with ID starting with 7c21c79db33019347909b7f2e26f28f8c43557d3100235c4c0fb2ec76267bce8 not found: ID does not exist"
Jul 29 21:37:51.782753 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.782733    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"]
Jul 29 21:37:51.786212 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.786191    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/auth-test-model-eb3087-kserve-85f8f7c65f-hgzm2"]
Jul 29 21:37:51.921694 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:51.921645    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs" podUID="28ef21d7-1e23-4ec2-85bc-87f3708ece31" containerName="main" containerID="cri-o://9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9" gracePeriod=45
Jul 29 21:37:52.181537 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.181510    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:37:52.221606 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221573    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-home\") pod \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") "
Jul 29 21:37:52.221776 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx48d\" (UniqueName: \"kubernetes.io/projected/28ef21d7-1e23-4ec2-85bc-87f3708ece31-kube-api-access-vx48d\") pod \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") "
Jul 29 21:37:52.221776 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221657    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tmp-dir\") pod \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") "
Jul 29 21:37:52.221776 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-dshm\") pod \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") "
Jul 29 21:37:52.221776 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tls-certs\") pod \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") "
Jul 29 21:37:52.221776 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221773    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-model-cache\") pod \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\" (UID: \"28ef21d7-1e23-4ec2-85bc-87f3708ece31\") "
Jul 29 21:37:52.221960 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221880    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-home" (OuterVolumeSpecName: "home") pod "28ef21d7-1e23-4ec2-85bc-87f3708ece31" (UID: "28ef21d7-1e23-4ec2-85bc-87f3708ece31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:52.221996 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.221968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "28ef21d7-1e23-4ec2-85bc-87f3708ece31" (UID: "28ef21d7-1e23-4ec2-85bc-87f3708ece31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:52.222097 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.222079    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-model-cache" (OuterVolumeSpecName: "model-cache") pod "28ef21d7-1e23-4ec2-85bc-87f3708ece31" (UID: "28ef21d7-1e23-4ec2-85bc-87f3708ece31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:52.222097 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.222089    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:52.222211 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.222107    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:52.224027 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.223996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-dshm" (OuterVolumeSpecName: "dshm") pod "28ef21d7-1e23-4ec2-85bc-87f3708ece31" (UID: "28ef21d7-1e23-4ec2-85bc-87f3708ece31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:37:52.224027 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.223994    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "28ef21d7-1e23-4ec2-85bc-87f3708ece31" (UID: "28ef21d7-1e23-4ec2-85bc-87f3708ece31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:37:52.224186 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.224089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ef21d7-1e23-4ec2-85bc-87f3708ece31-kube-api-access-vx48d" (OuterVolumeSpecName: "kube-api-access-vx48d") pod "28ef21d7-1e23-4ec2-85bc-87f3708ece31" (UID: "28ef21d7-1e23-4ec2-85bc-87f3708ece31"). InnerVolumeSpecName "kube-api-access-vx48d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:37:52.322951 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.322855    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx48d\" (UniqueName: \"kubernetes.io/projected/28ef21d7-1e23-4ec2-85bc-87f3708ece31-kube-api-access-vx48d\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:52.322951 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.322887    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:52.322951 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.322900    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28ef21d7-1e23-4ec2-85bc-87f3708ece31-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:52.322951 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.322913    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28ef21d7-1e23-4ec2-85bc-87f3708ece31-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:37:52.768736 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.768705    2579 generic.go:356] "Generic (PLEG): container finished" podID="28ef21d7-1e23-4ec2-85bc-87f3708ece31" containerID="9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9" exitCode=0
Jul 29 21:37:52.768736 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.768744    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"
Jul 29 21:37:52.768996 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.768771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs" event={"ID":"28ef21d7-1e23-4ec2-85bc-87f3708ece31","Type":"ContainerDied","Data":"9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9"}
Jul 29 21:37:52.768996 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.768801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs" event={"ID":"28ef21d7-1e23-4ec2-85bc-87f3708ece31","Type":"ContainerDied","Data":"495be55f90a3268d8cc14c0ae9ce3b1abc0247f09e15af25cf70d4170e87be95"}
Jul 29 21:37:52.768996 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.768816    2579 scope.go:117] "RemoveContainer" containerID="9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9"
Jul 29 21:37:52.778026 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.778009    2579 scope.go:117] "RemoveContainer" containerID="9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9"
Jul 29 21:37:52.778281 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:37:52.778264    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9\": container with ID starting with 9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9 not found: ID does not exist" containerID="9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9"
Jul 29 21:37:52.778344 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.778309    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9"} err="failed to get container status \"9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9\": rpc error: code = NotFound desc = could not find container \"9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9\": container with ID starting with 9ac815c083167720ec8a12e5a66f47a95c6b6912d0a718f414ceca80e96351f9 not found: ID does not exist"
Jul 29 21:37:52.789137 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.789108    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"]
Jul 29 21:37:52.792808 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:52.792787    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/auth-test-model-46bb38-kserve-b866b757f-8ffrs"]
Jul 29 21:37:53.255927 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:53.255889    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ef21d7-1e23-4ec2-85bc-87f3708ece31" path="/var/lib/kubelet/pods/28ef21d7-1e23-4ec2-85bc-87f3708ece31/volumes"
Jul 29 21:37:53.256320 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:53.256255    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d2c1d20-c260-402b-a779-8f171421f65c" path="/var/lib/kubelet/pods/2d2c1d20-c260-402b-a779-8f171421f65c/volumes"
Jul 29 21:37:56.366878 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.366845    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"]
Jul 29 21:37:56.367240 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.367193    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2c1d20-c260-402b-a779-8f171421f65c" containerName="main"
Jul 29 21:37:56.367240 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.367204    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2c1d20-c260-402b-a779-8f171421f65c" containerName="main"
Jul 29 21:37:56.367240 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.367212    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ef21d7-1e23-4ec2-85bc-87f3708ece31" containerName="main"
Jul 29 21:37:56.367240 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.367217    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ef21d7-1e23-4ec2-85bc-87f3708ece31" containerName="main"
Jul 29 21:37:56.367405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.367273    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ef21d7-1e23-4ec2-85bc-87f3708ece31" containerName="main"
Jul 29 21:37:56.367405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.367282    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2c1d20-c260-402b-a779-8f171421f65c" containerName="main"
Jul 29 21:37:56.371367 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.371351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.374777 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.374753    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"kube-root-ca.crt\""
Jul 29 21:37:56.374777 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.374767    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"default-dockercfg-mg2w8\""
Jul 29 21:37:56.374997 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.374780    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"sub-test-model-eb3087-kserve-self-signed-certs\""
Jul 29 21:37:56.375054 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.375039    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"openshift-service-ca.crt\""
Jul 29 21:37:56.381632 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.381613    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"]
Jul 29 21:37:56.455808 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.455774    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-home\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.455986 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.455815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-model-cache\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.455986 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.455832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj88k\" (UniqueName: \"kubernetes.io/projected/567fba27-45ec-48bc-8a83-8da7f7b10488-kube-api-access-cj88k\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.455986 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.455878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567fba27-45ec-48bc-8a83-8da7f7b10488-tls-certs\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.455986 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.455899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-dshm\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.455986 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.455972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-tmp-dir\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.556812 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.556774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-model-cache\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.556812 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.556813    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj88k\" (UniqueName: \"kubernetes.io/projected/567fba27-45ec-48bc-8a83-8da7f7b10488-kube-api-access-cj88k\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.557046 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.556848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567fba27-45ec-48bc-8a83-8da7f7b10488-tls-certs\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.557046 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.556877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-dshm\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.557046 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.556937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-tmp-dir\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.557046 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.556973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-home\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.557309 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.557212    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-model-cache\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.557369 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.557316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-home\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.557409 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.557365    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-tmp-dir\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.559029 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.559009    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-dshm\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.559308 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.559274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567fba27-45ec-48bc-8a83-8da7f7b10488-tls-certs\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.566009 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.565986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj88k\" (UniqueName: \"kubernetes.io/projected/567fba27-45ec-48bc-8a83-8da7f7b10488-kube-api-access-cj88k\") pod \"sub-test-model-eb3087-kserve-996d6478f-r947j\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") " pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.681393 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.681274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:37:56.815247 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:56.815226    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"]
Jul 29 21:37:56.817664 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:37:56.817636    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod567fba27_45ec_48bc_8a83_8da7f7b10488.slice/crio-215c1028de42720db8e03c230f0c7a49190898bf0b1c387984c765413b911dcf WatchSource:0}: Error finding container 215c1028de42720db8e03c230f0c7a49190898bf0b1c387984c765413b911dcf: Status 404 returned error can't find the container with id 215c1028de42720db8e03c230f0c7a49190898bf0b1c387984c765413b911dcf
Jul 29 21:37:57.801765 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:57.801727    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j" event={"ID":"567fba27-45ec-48bc-8a83-8da7f7b10488","Type":"ContainerStarted","Data":"f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923"}
Jul 29 21:37:57.801765 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:57.801772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j" event={"ID":"567fba27-45ec-48bc-8a83-8da7f7b10488","Type":"ContainerStarted","Data":"215c1028de42720db8e03c230f0c7a49190898bf0b1c387984c765413b911dcf"}
Jul 29 21:37:57.819917 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:37:57.819867    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j" podStartSLOduration=1.819852161 podStartE2EDuration="1.819852161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:37:57.819019846 +0000 UTC m=+3027.135508764" watchObservedRunningTime="2026-07-29 21:37:57.819852161 +0000 UTC m=+3027.136341076"
Jul 29 21:38:06.682023 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:06.681988    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:38:06.682518 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:06.682047    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:38:06.694373 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:06.694355    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:38:06.854257 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:06.854228    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:38:37.159592 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.159562    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"]
Jul 29 21:38:37.164255 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.164238    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.167474 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.167449    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"kube-root-ca.crt\""
Jul 29 21:38:37.167474 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.167463    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"default-dockercfg-pbrzs\""
Jul 29 21:38:37.167639 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.167485    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"sub-test-model-46bb38-kserve-self-signed-certs\""
Jul 29 21:38:37.167794 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.167778    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"openshift-service-ca.crt\""
Jul 29 21:38:37.172600 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.172582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"]
Jul 29 21:38:37.273747 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.273717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-tmp-dir\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.273923 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.273758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-dshm\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.273923 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.273852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78k4q\" (UniqueName: \"kubernetes.io/projected/2d90da5a-dd66-4602-8486-be4419e54b04-kube-api-access-78k4q\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.273923 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.273894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-model-cache\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.274102 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.273919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d90da5a-dd66-4602-8486-be4419e54b04-tls-certs\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.274102 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.273955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-home\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.374829 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.374803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78k4q\" (UniqueName: \"kubernetes.io/projected/2d90da5a-dd66-4602-8486-be4419e54b04-kube-api-access-78k4q\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.374936 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.374837    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-model-cache\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.374936 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.374855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d90da5a-dd66-4602-8486-be4419e54b04-tls-certs\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.374936 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.374878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-home\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.374936 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.374911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-tmp-dir\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.375144 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.374940    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-dshm\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.375248 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.375226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-model-cache\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.375340 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.375319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-tmp-dir\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.375384 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.375349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-home\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.377038 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.377021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-dshm\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.377281 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.377265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d90da5a-dd66-4602-8486-be4419e54b04-tls-certs\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.382151 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.382131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78k4q\" (UniqueName: \"kubernetes.io/projected/2d90da5a-dd66-4602-8486-be4419e54b04-kube-api-access-78k4q\") pod \"sub-test-model-46bb38-kserve-c49cb45f7-ln94c\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") " pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.473841 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.473793    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:37.598393 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.598363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"]
Jul 29 21:38:37.602197 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:38:37.602174    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d90da5a_dd66_4602_8486_be4419e54b04.slice/crio-606c8e267e778a15488c16c9f79a2987fa3334fae409a3c4c2bf31940144e308 WatchSource:0}: Error finding container 606c8e267e778a15488c16c9f79a2987fa3334fae409a3c4c2bf31940144e308: Status 404 returned error can't find the container with id 606c8e267e778a15488c16c9f79a2987fa3334fae409a3c4c2bf31940144e308
Jul 29 21:38:37.982772 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.982743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c" event={"ID":"2d90da5a-dd66-4602-8486-be4419e54b04","Type":"ContainerStarted","Data":"56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f"}
Jul 29 21:38:37.982998 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.982790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c" event={"ID":"2d90da5a-dd66-4602-8486-be4419e54b04","Type":"ContainerStarted","Data":"606c8e267e778a15488c16c9f79a2987fa3334fae409a3c4c2bf31940144e308"}
Jul 29 21:38:37.999921 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:37.999879    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c" podStartSLOduration=0.999866152 podStartE2EDuration="999.866152ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:38:37.999366337 +0000 UTC m=+3067.315855252" watchObservedRunningTime="2026-07-29 21:38:37.999866152 +0000 UTC m=+3067.316355066"
Jul 29 21:38:47.474379 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:47.474346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:47.474778 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:47.474392    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:47.486111 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:47.486084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:38:48.039926 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:38:48.039904    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:39:36.567658 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:36.567625    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"]
Jul 29 21:39:37.544698 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:37.544664    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"]
Jul 29 21:39:47.649564 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.649532    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"]
Jul 29 21:39:47.652599 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.652583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.655284 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.655264    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-eb3087\"/\"rate-test-model-eb3087-kserve-self-signed-certs\""
Jul 29 21:39:47.664115 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.664095    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"]
Jul 29 21:39:47.685445 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.685422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-tmp-dir\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.685544 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.685456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-model-cache\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.685544 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.685477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-home\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.685544 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.685498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwwdb\" (UniqueName: \"kubernetes.io/projected/161f9645-9867-435b-a47b-2772d8438eec-kube-api-access-qwwdb\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.685661 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.685551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161f9645-9867-435b-a47b-2772d8438eec-tls-certs\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.685661 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.685603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-dshm\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786260 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-model-cache\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786376 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786266    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-home\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786376 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786282    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwwdb\" (UniqueName: \"kubernetes.io/projected/161f9645-9867-435b-a47b-2772d8438eec-kube-api-access-qwwdb\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786376 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161f9645-9867-435b-a47b-2772d8438eec-tls-certs\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-dshm\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786494 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-tmp-dir\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786659 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-home\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-model-cache\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.786756 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.786744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-tmp-dir\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.788639 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.788612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-dshm\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.788811 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.788793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161f9645-9867-435b-a47b-2772d8438eec-tls-certs\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.794376 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.794356    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwwdb\" (UniqueName: \"kubernetes.io/projected/161f9645-9867-435b-a47b-2772d8438eec-kube-api-access-qwwdb\") pod \"rate-test-model-eb3087-kserve-f7c95dd95-57z8b\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") " pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:47.961333 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:47.961249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:48.089311 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:48.089274    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"]
Jul 29 21:39:48.091508 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:39:48.091479    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod161f9645_9867_435b_a47b_2772d8438eec.slice/crio-191e6211e473b1a30765b1b268970d6f3d906079618a876bf09a1951ffea31fb WatchSource:0}: Error finding container 191e6211e473b1a30765b1b268970d6f3d906079618a876bf09a1951ffea31fb: Status 404 returned error can't find the container with id 191e6211e473b1a30765b1b268970d6f3d906079618a876bf09a1951ffea31fb
Jul 29 21:39:48.309547 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:48.309463    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b" event={"ID":"161f9645-9867-435b-a47b-2772d8438eec","Type":"ContainerStarted","Data":"65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8"}
Jul 29 21:39:48.309547 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:48.309511    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b" event={"ID":"161f9645-9867-435b-a47b-2772d8438eec","Type":"ContainerStarted","Data":"191e6211e473b1a30765b1b268970d6f3d906079618a876bf09a1951ffea31fb"}
Jul 29 21:39:48.329562 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:48.329517    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b" podStartSLOduration=1.329504293 podStartE2EDuration="1.329504293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:39:48.327499972 +0000 UTC m=+3137.643988885" watchObservedRunningTime="2026-07-29 21:39:48.329504293 +0000 UTC m=+3137.645993207"
Jul 29 21:39:51.585689 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.585651    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j" podUID="567fba27-45ec-48bc-8a83-8da7f7b10488" containerName="main" containerID="cri-o://f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923" gracePeriod=45
Jul 29 21:39:51.830156 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.830133    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:39:51.922173 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj88k\" (UniqueName: \"kubernetes.io/projected/567fba27-45ec-48bc-8a83-8da7f7b10488-kube-api-access-cj88k\") pod \"567fba27-45ec-48bc-8a83-8da7f7b10488\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") "
Jul 29 21:39:51.922366 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922180    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-dshm\") pod \"567fba27-45ec-48bc-8a83-8da7f7b10488\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") "
Jul 29 21:39:51.922366 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922202    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-model-cache\") pod \"567fba27-45ec-48bc-8a83-8da7f7b10488\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") "
Jul 29 21:39:51.922366 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-home\") pod \"567fba27-45ec-48bc-8a83-8da7f7b10488\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") "
Jul 29 21:39:51.922536 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922388    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567fba27-45ec-48bc-8a83-8da7f7b10488-tls-certs\") pod \"567fba27-45ec-48bc-8a83-8da7f7b10488\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") "
Jul 29 21:39:51.922536 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922477    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-tmp-dir\") pod \"567fba27-45ec-48bc-8a83-8da7f7b10488\" (UID: \"567fba27-45ec-48bc-8a83-8da7f7b10488\") "
Jul 29 21:39:51.922536 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-model-cache" (OuterVolumeSpecName: "model-cache") pod "567fba27-45ec-48bc-8a83-8da7f7b10488" (UID: "567fba27-45ec-48bc-8a83-8da7f7b10488"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:51.922536 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922515    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-home" (OuterVolumeSpecName: "home") pod "567fba27-45ec-48bc-8a83-8da7f7b10488" (UID: "567fba27-45ec-48bc-8a83-8da7f7b10488"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:51.922741 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "567fba27-45ec-48bc-8a83-8da7f7b10488" (UID: "567fba27-45ec-48bc-8a83-8da7f7b10488"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:51.922839 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922824    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:51.922895 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922843    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:51.922895 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.922856    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:51.924357 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.924330    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-dshm" (OuterVolumeSpecName: "dshm") pod "567fba27-45ec-48bc-8a83-8da7f7b10488" (UID: "567fba27-45ec-48bc-8a83-8da7f7b10488"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:51.924357 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.924343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567fba27-45ec-48bc-8a83-8da7f7b10488-kube-api-access-cj88k" (OuterVolumeSpecName: "kube-api-access-cj88k") pod "567fba27-45ec-48bc-8a83-8da7f7b10488" (UID: "567fba27-45ec-48bc-8a83-8da7f7b10488"). InnerVolumeSpecName "kube-api-access-cj88k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:39:51.924491 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:51.924390    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/567fba27-45ec-48bc-8a83-8da7f7b10488-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "567fba27-45ec-48bc-8a83-8da7f7b10488" (UID: "567fba27-45ec-48bc-8a83-8da7f7b10488"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:39:52.023835 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.023806    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj88k\" (UniqueName: \"kubernetes.io/projected/567fba27-45ec-48bc-8a83-8da7f7b10488-kube-api-access-cj88k\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:52.023835 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.023834    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/567fba27-45ec-48bc-8a83-8da7f7b10488-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:52.024022 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.023846    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/567fba27-45ec-48bc-8a83-8da7f7b10488-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:52.329569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.329478    2579 generic.go:356] "Generic (PLEG): container finished" podID="567fba27-45ec-48bc-8a83-8da7f7b10488" containerID="f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923" exitCode=0
Jul 29 21:39:52.329569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.329521    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"
Jul 29 21:39:52.329569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.329538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j" event={"ID":"567fba27-45ec-48bc-8a83-8da7f7b10488","Type":"ContainerDied","Data":"f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923"}
Jul 29 21:39:52.329833 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.329578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j" event={"ID":"567fba27-45ec-48bc-8a83-8da7f7b10488","Type":"ContainerDied","Data":"215c1028de42720db8e03c230f0c7a49190898bf0b1c387984c765413b911dcf"}
Jul 29 21:39:52.329833 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.329599    2579 scope.go:117] "RemoveContainer" containerID="f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923"
Jul 29 21:39:52.338736 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.338720    2579 scope.go:117] "RemoveContainer" containerID="f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923"
Jul 29 21:39:52.338989 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:39:52.338974    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923\": container with ID starting with f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923 not found: ID does not exist" containerID="f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923"
Jul 29 21:39:52.339035 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.338997    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923"} err="failed to get container status \"f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923\": rpc error: code = NotFound desc = could not find container \"f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923\": container with ID starting with f6ad782247aea28b6f6085f8ffec9d02229509f89bfbf520f03feade86f5f923 not found: ID does not exist"
Jul 29 21:39:52.349882 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.349850    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"]
Jul 29 21:39:52.351707 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.351687    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/sub-test-model-eb3087-kserve-996d6478f-r947j"]
Jul 29 21:39:52.562477 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.562444    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c" podUID="2d90da5a-dd66-4602-8486-be4419e54b04" containerName="main" containerID="cri-o://56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f" gracePeriod=45
Jul 29 21:39:52.810417 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.809010    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:39:52.930743 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.930669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-dshm\") pod \"2d90da5a-dd66-4602-8486-be4419e54b04\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") "
Jul 29 21:39:52.930743 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.930707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d90da5a-dd66-4602-8486-be4419e54b04-tls-certs\") pod \"2d90da5a-dd66-4602-8486-be4419e54b04\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") "
Jul 29 21:39:52.930937 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.930770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-tmp-dir\") pod \"2d90da5a-dd66-4602-8486-be4419e54b04\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") "
Jul 29 21:39:52.930937 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.930878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-model-cache\") pod \"2d90da5a-dd66-4602-8486-be4419e54b04\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") "
Jul 29 21:39:52.931012 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.930960    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78k4q\" (UniqueName: \"kubernetes.io/projected/2d90da5a-dd66-4602-8486-be4419e54b04-kube-api-access-78k4q\") pod \"2d90da5a-dd66-4602-8486-be4419e54b04\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") "
Jul 29 21:39:52.931012 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.930993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d90da5a-dd66-4602-8486-be4419e54b04" (UID: "2d90da5a-dd66-4602-8486-be4419e54b04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:52.931012 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.931001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-home\") pod \"2d90da5a-dd66-4602-8486-be4419e54b04\" (UID: \"2d90da5a-dd66-4602-8486-be4419e54b04\") "
Jul 29 21:39:52.931179 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.931109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d90da5a-dd66-4602-8486-be4419e54b04" (UID: "2d90da5a-dd66-4602-8486-be4419e54b04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:52.931255 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.931238    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:52.931337 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.931270    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:52.931405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.931333    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-home" (OuterVolumeSpecName: "home") pod "2d90da5a-dd66-4602-8486-be4419e54b04" (UID: "2d90da5a-dd66-4602-8486-be4419e54b04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:52.932769 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.932737    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d90da5a-dd66-4602-8486-be4419e54b04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d90da5a-dd66-4602-8486-be4419e54b04" (UID: "2d90da5a-dd66-4602-8486-be4419e54b04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:39:52.932882 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.932857    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d90da5a-dd66-4602-8486-be4419e54b04-kube-api-access-78k4q" (OuterVolumeSpecName: "kube-api-access-78k4q") pod "2d90da5a-dd66-4602-8486-be4419e54b04" (UID: "2d90da5a-dd66-4602-8486-be4419e54b04"). InnerVolumeSpecName "kube-api-access-78k4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:39:52.932942 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:52.932891    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-dshm" (OuterVolumeSpecName: "dshm") pod "2d90da5a-dd66-4602-8486-be4419e54b04" (UID: "2d90da5a-dd66-4602-8486-be4419e54b04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:39:53.032499 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.032475    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78k4q\" (UniqueName: \"kubernetes.io/projected/2d90da5a-dd66-4602-8486-be4419e54b04-kube-api-access-78k4q\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:53.032499 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.032497    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:53.032625 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.032507    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d90da5a-dd66-4602-8486-be4419e54b04-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:53.032625 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.032514    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d90da5a-dd66-4602-8486-be4419e54b04-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:39:53.258755 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.258680    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="567fba27-45ec-48bc-8a83-8da7f7b10488" path="/var/lib/kubelet/pods/567fba27-45ec-48bc-8a83-8da7f7b10488/volumes"
Jul 29 21:39:53.336215 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.336181    2579 generic.go:356] "Generic (PLEG): container finished" podID="2d90da5a-dd66-4602-8486-be4419e54b04" containerID="56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f" exitCode=0
Jul 29 21:39:53.336409 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.336227    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"
Jul 29 21:39:53.336409 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.336231    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c" event={"ID":"2d90da5a-dd66-4602-8486-be4419e54b04","Type":"ContainerDied","Data":"56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f"}
Jul 29 21:39:53.336409 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.336329    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c" event={"ID":"2d90da5a-dd66-4602-8486-be4419e54b04","Type":"ContainerDied","Data":"606c8e267e778a15488c16c9f79a2987fa3334fae409a3c4c2bf31940144e308"}
Jul 29 21:39:53.336409 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.336352    2579 scope.go:117] "RemoveContainer" containerID="56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f"
Jul 29 21:39:53.344402 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.344387    2579 scope.go:117] "RemoveContainer" containerID="56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f"
Jul 29 21:39:53.344686 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:39:53.344667    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f\": container with ID starting with 56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f not found: ID does not exist" containerID="56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f"
Jul 29 21:39:53.344744 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.344693    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f"} err="failed to get container status \"56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f\": rpc error: code = NotFound desc = could not find container \"56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f\": container with ID starting with 56a30e0bd695dcc2794ff2412be0a74e70116d302fa3b77da4df180cb2e34e7f not found: ID does not exist"
Jul 29 21:39:53.352307 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.352270    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"]
Jul 29 21:39:53.355814 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:53.355787    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/sub-test-model-46bb38-kserve-c49cb45f7-ln94c"]
Jul 29 21:39:55.255383 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:55.255352    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d90da5a-dd66-4602-8486-be4419e54b04" path="/var/lib/kubelet/pods/2d90da5a-dd66-4602-8486-be4419e54b04/volumes"
Jul 29 21:39:57.961839 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:57.961791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:57.962318 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:57.961859    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:57.974261 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:57.974235    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:39:58.372970 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:39:58.372943    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:40:14.278984 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.278952    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"]
Jul 29 21:40:14.279464 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.279314    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="567fba27-45ec-48bc-8a83-8da7f7b10488" containerName="main"
Jul 29 21:40:14.279464 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.279351    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="567fba27-45ec-48bc-8a83-8da7f7b10488" containerName="main"
Jul 29 21:40:14.279464 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.279368    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d90da5a-dd66-4602-8486-be4419e54b04" containerName="main"
Jul 29 21:40:14.279464 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.279374    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d90da5a-dd66-4602-8486-be4419e54b04" containerName="main"
Jul 29 21:40:14.279464 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.279430    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d90da5a-dd66-4602-8486-be4419e54b04" containerName="main"
Jul 29 21:40:14.279464 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.279439    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="567fba27-45ec-48bc-8a83-8da7f7b10488" containerName="main"
Jul 29 21:40:14.282015 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.281997    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.285178 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.285158    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"rate-test-model-46bb38-kserve-self-signed-certs\""
Jul 29 21:40:14.285306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.285158    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"default-dockercfg-pbrzs\""
Jul 29 21:40:14.285306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.285160    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"kube-root-ca.crt\""
Jul 29 21:40:14.285306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.285158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-46bb38\"/\"openshift-service-ca.crt\""
Jul 29 21:40:14.292920 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.292902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"]
Jul 29 21:40:14.401698 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.401677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-home\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.401808 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.401708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs9f7\" (UniqueName: \"kubernetes.io/projected/4d5c289b-f1be-4371-b083-7baa629672a5-kube-api-access-bs9f7\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.401808 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.401728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-dshm\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.401808 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.401787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-model-cache\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.401928 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.401830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5c289b-f1be-4371-b083-7baa629672a5-tls-certs\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.401978 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.401920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-tmp-dir\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503230 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-model-cache\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503359 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5c289b-f1be-4371-b083-7baa629672a5-tls-certs\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503359 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-tmp-dir\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503359 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-home\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503526 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs9f7\" (UniqueName: \"kubernetes.io/projected/4d5c289b-f1be-4371-b083-7baa629672a5-kube-api-access-bs9f7\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503526 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-dshm\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503632 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-model-cache\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503690 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-tmp-dir\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.503754 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.503703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-home\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.505411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.505386    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-dshm\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.505653 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.505638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5c289b-f1be-4371-b083-7baa629672a5-tls-certs\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.512047 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.512030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs9f7\" (UniqueName: \"kubernetes.io/projected/4d5c289b-f1be-4371-b083-7baa629672a5-kube-api-access-bs9f7\") pod \"rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") " pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.591006 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.590953    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:14.743603 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:14.743581    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"]
Jul 29 21:40:14.745345 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:40:14.745318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d5c289b_f1be_4371_b083_7baa629672a5.slice/crio-24443aad5e29baf3b08cc0563c5671818fdf5845072e1447327417f238ec540f WatchSource:0}: Error finding container 24443aad5e29baf3b08cc0563c5671818fdf5845072e1447327417f238ec540f: Status 404 returned error can't find the container with id 24443aad5e29baf3b08cc0563c5671818fdf5845072e1447327417f238ec540f
Jul 29 21:40:15.442572 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:15.442529    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb" event={"ID":"4d5c289b-f1be-4371-b083-7baa629672a5","Type":"ContainerStarted","Data":"8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb"}
Jul 29 21:40:15.442572 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:15.442569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb" event={"ID":"4d5c289b-f1be-4371-b083-7baa629672a5","Type":"ContainerStarted","Data":"24443aad5e29baf3b08cc0563c5671818fdf5845072e1447327417f238ec540f"}
Jul 29 21:40:15.460073 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:15.460028    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb" podStartSLOduration=1.460013617 podStartE2EDuration="1.460013617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:40:15.459107233 +0000 UTC m=+3164.775596147" watchObservedRunningTime="2026-07-29 21:40:15.460013617 +0000 UTC m=+3164.776502551"
Jul 29 21:40:24.591101 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:24.591064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:24.591523 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:24.591113    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:24.602853 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:24.602830    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:40:25.498687 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:40:25.498653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:41:31.852133 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:31.852060    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"]
Jul 29 21:41:32.799423 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:32.799397    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"]
Jul 29 21:41:46.869739 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:46.869706    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b" podUID="161f9645-9867-435b-a47b-2772d8438eec" containerName="main" containerID="cri-o://65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8" gracePeriod=45
Jul 29 21:41:47.104898 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.104872    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:41:47.122258 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122199    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwwdb\" (UniqueName: \"kubernetes.io/projected/161f9645-9867-435b-a47b-2772d8438eec-kube-api-access-qwwdb\") pod \"161f9645-9867-435b-a47b-2772d8438eec\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") "
Jul 29 21:41:47.122258 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-home\") pod \"161f9645-9867-435b-a47b-2772d8438eec\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") "
Jul 29 21:41:47.122414 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-tmp-dir\") pod \"161f9645-9867-435b-a47b-2772d8438eec\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") "
Jul 29 21:41:47.122414 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122277    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-dshm\") pod \"161f9645-9867-435b-a47b-2772d8438eec\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") "
Jul 29 21:41:47.122414 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-model-cache\") pod \"161f9645-9867-435b-a47b-2772d8438eec\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") "
Jul 29 21:41:47.122414 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122406    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161f9645-9867-435b-a47b-2772d8438eec-tls-certs\") pod \"161f9645-9867-435b-a47b-2772d8438eec\" (UID: \"161f9645-9867-435b-a47b-2772d8438eec\") "
Jul 29 21:41:47.122616 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "161f9645-9867-435b-a47b-2772d8438eec" (UID: "161f9645-9867-435b-a47b-2772d8438eec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:47.122616 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-home" (OuterVolumeSpecName: "home") pod "161f9645-9867-435b-a47b-2772d8438eec" (UID: "161f9645-9867-435b-a47b-2772d8438eec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:47.122714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122656    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:47.122714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-model-cache" (OuterVolumeSpecName: "model-cache") pod "161f9645-9867-435b-a47b-2772d8438eec" (UID: "161f9645-9867-435b-a47b-2772d8438eec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:47.122714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.122675    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:47.124316 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.124271    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-dshm" (OuterVolumeSpecName: "dshm") pod "161f9645-9867-435b-a47b-2772d8438eec" (UID: "161f9645-9867-435b-a47b-2772d8438eec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:47.124316 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.124278    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/161f9645-9867-435b-a47b-2772d8438eec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "161f9645-9867-435b-a47b-2772d8438eec" (UID: "161f9645-9867-435b-a47b-2772d8438eec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:41:47.124316 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.124309    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161f9645-9867-435b-a47b-2772d8438eec-kube-api-access-qwwdb" (OuterVolumeSpecName: "kube-api-access-qwwdb") pod "161f9645-9867-435b-a47b-2772d8438eec" (UID: "161f9645-9867-435b-a47b-2772d8438eec"). InnerVolumeSpecName "kube-api-access-qwwdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:41:47.223668 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.223637    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:47.223668 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.223663    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/161f9645-9867-435b-a47b-2772d8438eec-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:47.223668 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.223672    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwwdb\" (UniqueName: \"kubernetes.io/projected/161f9645-9867-435b-a47b-2772d8438eec-kube-api-access-qwwdb\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:47.223880 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.223680    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/161f9645-9867-435b-a47b-2772d8438eec-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:47.817486 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.817449    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb" podUID="4d5c289b-f1be-4371-b083-7baa629672a5" containerName="main" containerID="cri-o://8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb" gracePeriod=45
Jul 29 21:41:47.874644 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.874612    2579 generic.go:356] "Generic (PLEG): container finished" podID="161f9645-9867-435b-a47b-2772d8438eec" containerID="65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8" exitCode=0
Jul 29 21:41:47.874945 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.874662    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"
Jul 29 21:41:47.874945 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.874661    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b" event={"ID":"161f9645-9867-435b-a47b-2772d8438eec","Type":"ContainerDied","Data":"65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8"}
Jul 29 21:41:47.874945 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.874781    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b" event={"ID":"161f9645-9867-435b-a47b-2772d8438eec","Type":"ContainerDied","Data":"191e6211e473b1a30765b1b268970d6f3d906079618a876bf09a1951ffea31fb"}
Jul 29 21:41:47.874945 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.874813    2579 scope.go:117] "RemoveContainer" containerID="65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8"
Jul 29 21:41:47.904279 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.904255    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"]
Jul 29 21:41:47.915887 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.915866    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-eb3087/rate-test-model-eb3087-kserve-f7c95dd95-57z8b"]
Jul 29 21:41:47.927969 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.927939    2579 scope.go:117] "RemoveContainer" containerID="65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8"
Jul 29 21:41:47.928230 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:41:47.928212    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8\": container with ID starting with 65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8 not found: ID does not exist" containerID="65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8"
Jul 29 21:41:47.928316 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:47.928236    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8"} err="failed to get container status \"65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8\": rpc error: code = NotFound desc = could not find container \"65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8\": container with ID starting with 65fea4d68dd0ccde79b61f97d15157dd37720a9edb9faa9340d07cb21f88d6e8 not found: ID does not exist"
Jul 29 21:41:48.054333 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.054313    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:41:48.131582 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131559    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-tmp-dir\") pod \"4d5c289b-f1be-4371-b083-7baa629672a5\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") "
Jul 29 21:41:48.131710 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131622    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5c289b-f1be-4371-b083-7baa629672a5-tls-certs\") pod \"4d5c289b-f1be-4371-b083-7baa629672a5\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") "
Jul 29 21:41:48.131710 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-home\") pod \"4d5c289b-f1be-4371-b083-7baa629672a5\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") "
Jul 29 21:41:48.131710 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-dshm\") pod \"4d5c289b-f1be-4371-b083-7baa629672a5\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") "
Jul 29 21:41:48.131710 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bs9f7\" (UniqueName: \"kubernetes.io/projected/4d5c289b-f1be-4371-b083-7baa629672a5-kube-api-access-bs9f7\") pod \"4d5c289b-f1be-4371-b083-7baa629672a5\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") "
Jul 29 21:41:48.131923 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131716    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-model-cache\") pod \"4d5c289b-f1be-4371-b083-7baa629672a5\" (UID: \"4d5c289b-f1be-4371-b083-7baa629672a5\") "
Jul 29 21:41:48.131923 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4d5c289b-f1be-4371-b083-7baa629672a5" (UID: "4d5c289b-f1be-4371-b083-7baa629672a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:48.131923 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.131870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-home" (OuterVolumeSpecName: "home") pod "4d5c289b-f1be-4371-b083-7baa629672a5" (UID: "4d5c289b-f1be-4371-b083-7baa629672a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:48.132066 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.132031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d5c289b-f1be-4371-b083-7baa629672a5" (UID: "4d5c289b-f1be-4371-b083-7baa629672a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:48.133715 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.133687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-dshm" (OuterVolumeSpecName: "dshm") pod "4d5c289b-f1be-4371-b083-7baa629672a5" (UID: "4d5c289b-f1be-4371-b083-7baa629672a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:41:48.133804 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.133734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5c289b-f1be-4371-b083-7baa629672a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4d5c289b-f1be-4371-b083-7baa629672a5" (UID: "4d5c289b-f1be-4371-b083-7baa629672a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:41:48.133804 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.133772    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5c289b-f1be-4371-b083-7baa629672a5-kube-api-access-bs9f7" (OuterVolumeSpecName: "kube-api-access-bs9f7") pod "4d5c289b-f1be-4371-b083-7baa629672a5" (UID: "4d5c289b-f1be-4371-b083-7baa629672a5"). InnerVolumeSpecName "kube-api-access-bs9f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:41:48.232413 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.232391    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:48.232413 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.232412    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs9f7\" (UniqueName: \"kubernetes.io/projected/4d5c289b-f1be-4371-b083-7baa629672a5-kube-api-access-bs9f7\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:48.232549 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.232420    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:48.232549 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.232428    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:48.232549 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.232437    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d5c289b-f1be-4371-b083-7baa629672a5-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:48.232549 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.232444    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d5c289b-f1be-4371-b083-7baa629672a5-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:48.527318 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.527233    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7477-139989194132480"]
Jul 29 21:41:48.527583 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.527571    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d5c289b-f1be-4371-b083-7baa629672a5" containerName="main"
Jul 29 21:41:48.527639 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.527585    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d5c289b-f1be-4371-b083-7baa629672a5" containerName="main"
Jul 29 21:41:48.527639 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.527593    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161f9645-9867-435b-a47b-2772d8438eec" containerName="main"
Jul 29 21:41:48.527639 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.527598    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="161f9645-9867-435b-a47b-2772d8438eec" containerName="main"
Jul 29 21:41:48.527735 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.527647    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="161f9645-9867-435b-a47b-2772d8438eec" containerName="main"
Jul 29 21:41:48.527735 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.527657    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d5c289b-f1be-4371-b083-7baa629672a5" containerName="main"
Jul 29 21:41:48.530584 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.530570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480"
Jul 29 21:41:48.533176 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.533158    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zmhw5\""
Jul 29 21:41:48.542362 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.542338    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"]
Jul 29 21:41:48.635180 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.635154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl27p\" (UniqueName: \"kubernetes.io/projected/e99cb06c-8c3f-4559-8723-3de79060657e-kube-api-access-xl27p\") pod \"test-curl-7477-139989194132480\" (UID: \"e99cb06c-8c3f-4559-8723-3de79060657e\") " pod="opendatahub/test-curl-7477-139989194132480"
Jul 29 21:41:48.735695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.735673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl27p\" (UniqueName: \"kubernetes.io/projected/e99cb06c-8c3f-4559-8723-3de79060657e-kube-api-access-xl27p\") pod \"test-curl-7477-139989194132480\" (UID: \"e99cb06c-8c3f-4559-8723-3de79060657e\") " pod="opendatahub/test-curl-7477-139989194132480"
Jul 29 21:41:48.744839 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.744815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl27p\" (UniqueName: \"kubernetes.io/projected/e99cb06c-8c3f-4559-8723-3de79060657e-kube-api-access-xl27p\") pod \"test-curl-7477-139989194132480\" (UID: \"e99cb06c-8c3f-4559-8723-3de79060657e\") " pod="opendatahub/test-curl-7477-139989194132480"
Jul 29 21:41:48.839682 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.839619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480"
Jul 29 21:41:48.883144 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.883117    2579 generic.go:356] "Generic (PLEG): container finished" podID="4d5c289b-f1be-4371-b083-7baa629672a5" containerID="8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb" exitCode=0
Jul 29 21:41:48.883644 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.883163    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"
Jul 29 21:41:48.883644 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.883179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb" event={"ID":"4d5c289b-f1be-4371-b083-7baa629672a5","Type":"ContainerDied","Data":"8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb"}
Jul 29 21:41:48.883644 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.883208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb" event={"ID":"4d5c289b-f1be-4371-b083-7baa629672a5","Type":"ContainerDied","Data":"24443aad5e29baf3b08cc0563c5671818fdf5845072e1447327417f238ec540f"}
Jul 29 21:41:48.883644 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.883224    2579 scope.go:117] "RemoveContainer" containerID="8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb"
Jul 29 21:41:48.899984 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.899961    2579 scope.go:117] "RemoveContainer" containerID="8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb"
Jul 29 21:41:48.900349 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:41:48.900322    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb\": container with ID starting with 8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb not found: ID does not exist" containerID="8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb"
Jul 29 21:41:48.900444 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.900359    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb"} err="failed to get container status \"8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb\": rpc error: code = NotFound desc = could not find container \"8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb\": container with ID starting with 8202d04a68e977d71d4177ac0a30c6bc7320743774bfa0041c9bb0c0784b82fb not found: ID does not exist"
Jul 29 21:41:48.912784 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.912762    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"]
Jul 29 21:41:48.917511 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.917492    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-46bb38/rate-test-model-46bb38-kserve-5d9fb7bb87-k29tb"]
Jul 29 21:41:48.976596 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.976568    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"]
Jul 29 21:41:48.979621 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:41:48.979595    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode99cb06c_8c3f_4559_8723_3de79060657e.slice/crio-39b1461e025a84a9d26223f276556fd29c1d27bd086b8ebd2c28cd54ac82d2f7 WatchSource:0}: Error finding container 39b1461e025a84a9d26223f276556fd29c1d27bd086b8ebd2c28cd54ac82d2f7: Status 404 returned error can't find the container with id 39b1461e025a84a9d26223f276556fd29c1d27bd086b8ebd2c28cd54ac82d2f7
Jul 29 21:41:48.980824 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:48.980808    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:41:49.255016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:49.254980    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161f9645-9867-435b-a47b-2772d8438eec" path="/var/lib/kubelet/pods/161f9645-9867-435b-a47b-2772d8438eec/volumes"
Jul 29 21:41:49.255422 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:49.255407    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d5c289b-f1be-4371-b083-7baa629672a5" path="/var/lib/kubelet/pods/4d5c289b-f1be-4371-b083-7baa629672a5/volumes"
Jul 29 21:41:49.890909 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:49.890815    2579 generic.go:356] "Generic (PLEG): container finished" podID="e99cb06c-8c3f-4559-8723-3de79060657e" containerID="d8c69146c77e44db9ee52ea14d36e9c626933341d3850211bd419947557ad716" exitCode=0
Jul 29 21:41:49.890909 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:49.890856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194132480" event={"ID":"e99cb06c-8c3f-4559-8723-3de79060657e","Type":"ContainerDied","Data":"d8c69146c77e44db9ee52ea14d36e9c626933341d3850211bd419947557ad716"}
Jul 29 21:41:49.890909 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:49.890879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7477-139989194132480" event={"ID":"e99cb06c-8c3f-4559-8723-3de79060657e","Type":"ContainerStarted","Data":"39b1461e025a84a9d26223f276556fd29c1d27bd086b8ebd2c28cd54ac82d2f7"}
Jul 29 21:41:51.027938 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.027910    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480"
Jul 29 21:41:51.053505 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.053471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl27p\" (UniqueName: \"kubernetes.io/projected/e99cb06c-8c3f-4559-8723-3de79060657e-kube-api-access-xl27p\") pod \"e99cb06c-8c3f-4559-8723-3de79060657e\" (UID: \"e99cb06c-8c3f-4559-8723-3de79060657e\") "
Jul 29 21:41:51.055961 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.055931    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99cb06c-8c3f-4559-8723-3de79060657e-kube-api-access-xl27p" (OuterVolumeSpecName: "kube-api-access-xl27p") pod "e99cb06c-8c3f-4559-8723-3de79060657e" (UID: "e99cb06c-8c3f-4559-8723-3de79060657e"). InnerVolumeSpecName "kube-api-access-xl27p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:41:51.092937 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.092913    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7477-139989194132480_e99cb06c-8c3f-4559-8723-3de79060657e/test-curl-7477-139989194132480/0.log"
Jul 29 21:41:51.123128 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.123096    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"]
Jul 29 21:41:51.128489 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.128468    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7477-139989194132480"]
Jul 29 21:41:51.154647 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.154595    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl27p\" (UniqueName: \"kubernetes.io/projected/e99cb06c-8c3f-4559-8723-3de79060657e-kube-api-access-xl27p\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:41:51.256404 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.256373    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99cb06c-8c3f-4559-8723-3de79060657e" path="/var/lib/kubelet/pods/e99cb06c-8c3f-4559-8723-3de79060657e/volumes"
Jul 29 21:41:51.901463 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.901435    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7477-139989194132480"
Jul 29 21:41:51.901641 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:41:51.901486    2579 scope.go:117] "RemoveContainer" containerID="d8c69146c77e44db9ee52ea14d36e9c626933341d3850211bd419947557ad716"
Jul 29 21:42:19.658447 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.658416    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sz2lf"]
Jul 29 21:42:19.658894 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.658729    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99cb06c-8c3f-4559-8723-3de79060657e" containerName="test-curl-7477-139989194132480"
Jul 29 21:42:19.658894 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.658740    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99cb06c-8c3f-4559-8723-3de79060657e" containerName="test-curl-7477-139989194132480"
Jul 29 21:42:19.658894 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.658813    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99cb06c-8c3f-4559-8723-3de79060657e" containerName="test-curl-7477-139989194132480"
Jul 29 21:42:19.661476 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.661457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:19.664404 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.664386    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-9sfb2\""
Jul 29 21:42:19.675427 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.675410    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sz2lf"]
Jul 29 21:42:19.767913 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.767885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4grm\" (UniqueName: \"kubernetes.io/projected/1d4b7b1b-141e-408c-a79d-098fb558ee6b-kube-api-access-r4grm\") pod \"kuadrant-operator-catalog-sz2lf\" (UID: \"1d4b7b1b-141e-408c-a79d-098fb558ee6b\") " pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:19.868194 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.868170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4grm\" (UniqueName: \"kubernetes.io/projected/1d4b7b1b-141e-408c-a79d-098fb558ee6b-kube-api-access-r4grm\") pod \"kuadrant-operator-catalog-sz2lf\" (UID: \"1d4b7b1b-141e-408c-a79d-098fb558ee6b\") " pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:19.876440 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.876415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4grm\" (UniqueName: \"kubernetes.io/projected/1d4b7b1b-141e-408c-a79d-098fb558ee6b-kube-api-access-r4grm\") pod \"kuadrant-operator-catalog-sz2lf\" (UID: \"1d4b7b1b-141e-408c-a79d-098fb558ee6b\") " pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:19.970710 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:19.970657    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:20.120723 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:20.120693    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sz2lf"]
Jul 29 21:42:20.122483 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:42:20.122458    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d4b7b1b_141e_408c_a79d_098fb558ee6b.slice/crio-56a62de4fc6f23199674a5310a68c3e4e0e0dc0f63c43f2dab426fea52e435d1 WatchSource:0}: Error finding container 56a62de4fc6f23199674a5310a68c3e4e0e0dc0f63c43f2dab426fea52e435d1: Status 404 returned error can't find the container with id 56a62de4fc6f23199674a5310a68c3e4e0e0dc0f63c43f2dab426fea52e435d1
Jul 29 21:42:21.040997 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.040964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf" event={"ID":"1d4b7b1b-141e-408c-a79d-098fb558ee6b","Type":"ContainerStarted","Data":"56a62de4fc6f23199674a5310a68c3e4e0e0dc0f63c43f2dab426fea52e435d1"}
Jul 29 21:42:21.514484 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.514451    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"]
Jul 29 21:42:21.518066 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.518046    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.525752 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.525732    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 21:42:21.527946 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.527927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-00731e-openshift-default-dockercfg-shthc\""
Jul 29 21:42:21.543356 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.543330    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"]
Jul 29 21:42:21.583935 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.583909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-certs\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584062 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.583951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-data\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584062 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.583988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-envoy\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584062 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.584013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882195f-a528-4807-a3d6-15c6e92f2578-istio-podinfo\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584062 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.584056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-credential-socket\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584259 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.584110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-socket\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584259 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.584133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgfh9\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-kube-api-access-bgfh9\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584259 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.584223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882195f-a528-4807-a3d6-15c6e92f2578-istiod-ca-cert\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.584432 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.584263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-istio-token\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.685747 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685714    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-socket\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.685895 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685753    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgfh9\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-kube-api-access-bgfh9\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.685895 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882195f-a528-4807-a3d6-15c6e92f2578-istiod-ca-cert\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.685895 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-istio-token\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.685895 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-certs\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686130 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-data\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686130 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685952    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-envoy\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686130 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882195f-a528-4807-a3d6-15c6e92f2578-istio-podinfo\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686130 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.685998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-credential-socket\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686376 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.686133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-socket\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686376 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.686279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-data\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686553 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.686533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-credential-socket\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686665 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.686635    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-certs\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.686665 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.686654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882195f-a528-4807-a3d6-15c6e92f2578-istiod-ca-cert\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.688725 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.688698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-envoy\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.689055 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.689031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882195f-a528-4807-a3d6-15c6e92f2578-istio-podinfo\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.701628 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.701586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-istio-token\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.703779 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.703753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgfh9\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-kube-api-access-bgfh9\") pod \"e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") " pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.830663 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.830591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:21.979024 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:21.978995    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"]
Jul 29 21:42:21.982261 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:42:21.982225    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7882195f_a528_4807_a3d6_15c6e92f2578.slice/crio-b5e44582951c9a5f3809985f18b4308b4383d27f634434d6e35dcf1344897b1d WatchSource:0}: Error finding container b5e44582951c9a5f3809985f18b4308b4383d27f634434d6e35dcf1344897b1d: Status 404 returned error can't find the container with id b5e44582951c9a5f3809985f18b4308b4383d27f634434d6e35dcf1344897b1d
Jul 29 21:42:22.046774 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:22.046743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" event={"ID":"7882195f-a528-4807-a3d6-15c6e92f2578","Type":"ContainerStarted","Data":"b5e44582951c9a5f3809985f18b4308b4383d27f634434d6e35dcf1344897b1d"}
Jul 29 21:42:22.048353 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:22.048332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf" event={"ID":"1d4b7b1b-141e-408c-a79d-098fb558ee6b","Type":"ContainerStarted","Data":"7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9"}
Jul 29 21:42:22.078664 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:22.078623    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf" podStartSLOduration=1.3145509149999999 podStartE2EDuration="3.078609108s" totalImagesPullingTime="1.764058193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:42:22.078505972 +0000 UTC m=+3291.394994886" watchObservedRunningTime="2026-07-29 21:42:22.078609108 +0000 UTC m=+3291.395098020"
Jul 29 21:42:29.971459 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:29.971429    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:29.971889 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:29.971475    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:29.992876 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:29.992852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:30.106545 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:30.106523    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:30.252692 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:30.252618    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sz2lf"]
Jul 29 21:42:32.095632 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:32.095568    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf" podUID="1d4b7b1b-141e-408c-a79d-098fb558ee6b" containerName="registry-server" containerID="cri-o://7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9" gracePeriod=2
Jul 29 21:42:32.986888 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:32.986852    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:42:32.986980 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:32.986925    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:42:32.986980 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:32.986955    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:42:33.077946 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.077924    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:33.102476 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.102440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" event={"ID":"7882195f-a528-4807-a3d6-15c6e92f2578","Type":"ContainerStarted","Data":"944d22c0af337d6b437b12da83b6dad94d68dea8c30b3c4c092f4157c1e7d06a"}
Jul 29 21:42:33.104551 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.104525    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d4b7b1b-141e-408c-a79d-098fb558ee6b" containerID="7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9" exitCode=0
Jul 29 21:42:33.104688 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.104559    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf"
Jul 29 21:42:33.104688 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.104561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf" event={"ID":"1d4b7b1b-141e-408c-a79d-098fb558ee6b","Type":"ContainerDied","Data":"7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9"}
Jul 29 21:42:33.104802 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.104691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sz2lf" event={"ID":"1d4b7b1b-141e-408c-a79d-098fb558ee6b","Type":"ContainerDied","Data":"56a62de4fc6f23199674a5310a68c3e4e0e0dc0f63c43f2dab426fea52e435d1"}
Jul 29 21:42:33.104802 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.104721    2579 scope.go:117] "RemoveContainer" containerID="7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9"
Jul 29 21:42:33.115706 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.115680    2579 scope.go:117] "RemoveContainer" containerID="7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9"
Jul 29 21:42:33.116004 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:42:33.115984    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9\": container with ID starting with 7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9 not found: ID does not exist" containerID="7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9"
Jul 29 21:42:33.116086 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.116018    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9"} err="failed to get container status \"7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9\": rpc error: code = NotFound desc = could not find container \"7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9\": container with ID starting with 7ba81fd9ebbf7e86dd24723a57f01c9bc59f9ea36e2c8959a5fa5c498e3d80c9 not found: ID does not exist"
Jul 29 21:42:33.135544 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.135491    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" podStartSLOduration=1.133523131 podStartE2EDuration="12.135471843s" totalImagesPullingTime="11.001948712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:42:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:42:33.133496089 +0000 UTC m=+3302.449985000" watchObservedRunningTime="2026-07-29 21:42:33.135471843 +0000 UTC m=+3302.451960758"
Jul 29 21:42:33.180256 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.180192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4grm\" (UniqueName: \"kubernetes.io/projected/1d4b7b1b-141e-408c-a79d-098fb558ee6b-kube-api-access-r4grm\") pod \"1d4b7b1b-141e-408c-a79d-098fb558ee6b\" (UID: \"1d4b7b1b-141e-408c-a79d-098fb558ee6b\") "
Jul 29 21:42:33.182320 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.182273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4b7b1b-141e-408c-a79d-098fb558ee6b-kube-api-access-r4grm" (OuterVolumeSpecName: "kube-api-access-r4grm") pod "1d4b7b1b-141e-408c-a79d-098fb558ee6b" (UID: "1d4b7b1b-141e-408c-a79d-098fb558ee6b"). InnerVolumeSpecName "kube-api-access-r4grm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:42:33.281741 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.281716    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4grm\" (UniqueName: \"kubernetes.io/projected/1d4b7b1b-141e-408c-a79d-098fb558ee6b-kube-api-access-r4grm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:42:33.427019 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.426990    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sz2lf"]
Jul 29 21:42:33.430534 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.430469    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sz2lf"]
Jul 29 21:42:33.830775 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:33.830698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:34.835159 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:34.835133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:35.115330 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:35.115303    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:35.120158 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:35.120137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:35.255908 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:35.255870    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d4b7b1b-141e-408c-a79d-098fb558ee6b" path="/var/lib/kubelet/pods/1d4b7b1b-141e-408c-a79d-098fb558ee6b/volumes"
Jul 29 21:42:50.321652 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:50.321381    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:50426->10.133.0.51:15021: read: connection reset by peer" start-of-body=
Jul 29 21:42:50.321652 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:50.321446    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": read tcp 10.133.0.2:50426->10.133.0.51:15021: read: connection reset by peer"
Jul 29 21:42:51.197118 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:51.197083    2579 generic.go:356] "Generic (PLEG): container finished" podID="7882195f-a528-4807-a3d6-15c6e92f2578" containerID="944d22c0af337d6b437b12da83b6dad94d68dea8c30b3c4c092f4157c1e7d06a" exitCode=0
Jul 29 21:42:51.197324 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:51.197161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" event={"ID":"7882195f-a528-4807-a3d6-15c6e92f2578","Type":"ContainerDied","Data":"944d22c0af337d6b437b12da83b6dad94d68dea8c30b3c4c092f4157c1e7d06a"}
Jul 29 21:42:51.197574 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:51.197561    2579 scope.go:117] "RemoveContainer" containerID="944d22c0af337d6b437b12da83b6dad94d68dea8c30b3c4c092f4157c1e7d06a"
Jul 29 21:42:51.199578 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:51.199547    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:42:51.199682 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:51.199634    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:42:51.199745 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:51.199683    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:42:52.204125 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:52.204093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" event={"ID":"7882195f-a528-4807-a3d6-15c6e92f2578","Type":"ContainerStarted","Data":"c778f9fbd7f22da4f4aa9d501e27e0ba5c4831da2820ed449c8827bb340e3b6d"}
Jul 29 21:42:52.831439 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:52.831349    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:52.831599 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:52.831575    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:42:52.831650 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:52.831622    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 29 21:42:53.831165 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:53.831130    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:42:53.831582 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:53.831188    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 29 21:42:54.831128 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:54.831085    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:42:54.831351 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:54.831146    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Jul 29 21:42:55.834179 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:55.834139    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:55.834555 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:55.834487    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:42:55.835171 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:42:55.835155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:43:22.847411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.844936    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"]
Jul 29 21:43:22.847411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.845692    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4b7b1b-141e-408c-a79d-098fb558ee6b" containerName="registry-server"
Jul 29 21:43:22.847411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.845711    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4b7b1b-141e-408c-a79d-098fb558ee6b" containerName="registry-server"
Jul 29 21:43:22.847411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.845896    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4b7b1b-141e-408c-a79d-098fb558ee6b" containerName="registry-server"
Jul 29 21:43:22.851075 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.851049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:22.854318 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.854275    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f2f2fe\"/\"ipp-route-f2f2fe-kserve-self-signed-certs\""
Jul 29 21:43:22.854318 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.854281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-f2f2fe\"/\"default-dockercfg-r46t7\""
Jul 29 21:43:22.854509 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.854281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f2f2fe\"/\"kube-root-ca.crt\""
Jul 29 21:43:22.854509 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.854275    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-f2f2fe\"/\"openshift-service-ca.crt\""
Jul 29 21:43:22.860534 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.860515    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"]
Jul 29 21:43:22.982777 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.982751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-tmp-dir\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:22.982918 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.982806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-home\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:22.982918 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.982836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-model-cache\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:22.982918 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.982867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-dshm\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:22.982918 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.982904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr52c\" (UniqueName: \"kubernetes.io/projected/3c65614d-7089-4ba0-9547-6fe9a3823c03-kube-api-access-wr52c\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:22.983064 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:22.982951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c65614d-7089-4ba0-9547-6fe9a3823c03-tls-certs\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.083886 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.083862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr52c\" (UniqueName: \"kubernetes.io/projected/3c65614d-7089-4ba0-9547-6fe9a3823c03-kube-api-access-wr52c\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084029 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.083903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c65614d-7089-4ba0-9547-6fe9a3823c03-tls-certs\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084029 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.083934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-tmp-dir\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084029 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.083962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-home\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084029 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.083998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-model-cache\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084029 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.084021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-dshm\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084433 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.084409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-tmp-dir\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084533 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.084419    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-model-cache\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.084533 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.084511    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-home\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.086140 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.086112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-dshm\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.086311 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.086278    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c65614d-7089-4ba0-9547-6fe9a3823c03-tls-certs\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.092940 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.092917    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr52c\" (UniqueName: \"kubernetes.io/projected/3c65614d-7089-4ba0-9547-6fe9a3823c03-kube-api-access-wr52c\") pod \"ipp-route-f2f2fe-kserve-788f56447f-xxxg4\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") " pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.160709 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.160649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:23.290510 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.290481    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"]
Jul 29 21:43:23.292961 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:43:23.292935    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c65614d_7089_4ba0_9547_6fe9a3823c03.slice/crio-1c10e58d8f9e660510e4873fd279b72e4cea15b32824673ce7387bc399f359e4 WatchSource:0}: Error finding container 1c10e58d8f9e660510e4873fd279b72e4cea15b32824673ce7387bc399f359e4: Status 404 returned error can't find the container with id 1c10e58d8f9e660510e4873fd279b72e4cea15b32824673ce7387bc399f359e4
Jul 29 21:43:23.344738 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:23.344710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4" event={"ID":"3c65614d-7089-4ba0-9547-6fe9a3823c03","Type":"ContainerStarted","Data":"1c10e58d8f9e660510e4873fd279b72e4cea15b32824673ce7387bc399f359e4"}
Jul 29 21:43:24.351248 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:24.351208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4" event={"ID":"3c65614d-7089-4ba0-9547-6fe9a3823c03","Type":"ContainerStarted","Data":"61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd"}
Jul 29 21:43:24.371411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:24.371366    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4" podStartSLOduration=2.371343246 podStartE2EDuration="2.371343246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:43:24.370075302 +0000 UTC m=+3353.686564215" watchObservedRunningTime="2026-07-29 21:43:24.371343246 +0000 UTC m=+3353.687832168"
Jul 29 21:43:33.161070 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:33.161034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:33.161489 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:33.161095    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:33.173242 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:33.173220    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:43:33.402408 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:43:33.402379    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:44:05.391401 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.391357    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"]
Jul 29 21:44:05.394792 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.394775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.397259 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.397238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-50783d-openshift-default-dockercfg-hkhmh\""
Jul 29 21:44:05.411481 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.411460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"]
Jul 29 21:44:05.423481 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-podinfo\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-certs\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-token\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9cs8\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-kube-api-access-q9cs8\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423674 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-data\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423674 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-credential-socket\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423743 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-envoy\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423743 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istiod-ca-cert\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.423743 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.423732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-socket\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.524626 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-envoy\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.524789 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istiod-ca-cert\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.524789 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-socket\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.524789 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-podinfo\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.524789 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-certs\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.524789 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-token\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.524789 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9cs8\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-kube-api-access-q9cs8\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.525106 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524812    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-data\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.525106 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.524855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-credential-socket\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.525216 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.525100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-socket\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.525216 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.525124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-certs\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.525216 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.525198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-credential-socket\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.525396 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.525380    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-data\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.525434 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.525407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istiod-ca-cert\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.527361 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.527342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-envoy\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.527540 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.527522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-podinfo\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.532668 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.532646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9cs8\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-kube-api-access-q9cs8\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.532784 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.532771    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-token\") pod \"e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") " pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.704446 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.704379    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:05.839322 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.839284    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"]
Jul 29 21:44:05.841760 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:44:05.841736    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff57fac_cd7a_45a7_9ef5_f98448c1a5c7.slice/crio-185c4555f2aa2103caea9f7666f34c1127ee7ad07cb6c297031c4bb69e2b5cb8 WatchSource:0}: Error finding container 185c4555f2aa2103caea9f7666f34c1127ee7ad07cb6c297031c4bb69e2b5cb8: Status 404 returned error can't find the container with id 185c4555f2aa2103caea9f7666f34c1127ee7ad07cb6c297031c4bb69e2b5cb8
Jul 29 21:44:05.843958 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.843930    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:44:05.844026 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.843994    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:44:05.844026 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:05.844020    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:44:06.546551 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:06.546522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" event={"ID":"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7","Type":"ContainerStarted","Data":"822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45"}
Jul 29 21:44:06.546920 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:06.546569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" event={"ID":"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7","Type":"ContainerStarted","Data":"185c4555f2aa2103caea9f7666f34c1127ee7ad07cb6c297031c4bb69e2b5cb8"}
Jul 29 21:44:06.579152 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:06.579089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" podStartSLOduration=1.579071909 podStartE2EDuration="1.579071909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:44:06.57816236 +0000 UTC m=+3395.894651274" watchObservedRunningTime="2026-07-29 21:44:06.579071909 +0000 UTC m=+3395.895560823"
Jul 29 21:44:06.705062 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:06.705029    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:06.709429 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:06.709406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:07.551598 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:07.551565    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:07.552365 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:07.552344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:29.066441 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.066357    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"]
Jul 29 21:44:29.068889 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.068874    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"
Jul 29 21:44:29.071158 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.071137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:44:29.071276 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.071176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:44:29.077022 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.077000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"]
Jul 29 21:44:29.126830 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.126810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d613b306-f1b4-4080-a7ac-875908b89afb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk\" (UID: \"d613b306-f1b4-4080-a7ac-875908b89afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"
Jul 29 21:44:29.228129 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.228105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d613b306-f1b4-4080-a7ac-875908b89afb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk\" (UID: \"d613b306-f1b4-4080-a7ac-875908b89afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"
Jul 29 21:44:29.228846 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.228827    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d613b306-f1b4-4080-a7ac-875908b89afb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk\" (UID: \"d613b306-f1b4-4080-a7ac-875908b89afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"
Jul 29 21:44:29.380092 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.380067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"
Jul 29 21:44:29.507049 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.507025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"]
Jul 29 21:44:29.509090 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:44:29.509062    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd613b306_f1b4_4080_a7ac_875908b89afb.slice/crio-49d9092d432dcbf3ed9e28533aa37082ebb5b1222b45ad6fbc90dbbae01ade6a WatchSource:0}: Error finding container 49d9092d432dcbf3ed9e28533aa37082ebb5b1222b45ad6fbc90dbbae01ade6a: Status 404 returned error can't find the container with id 49d9092d432dcbf3ed9e28533aa37082ebb5b1222b45ad6fbc90dbbae01ade6a
Jul 29 21:44:29.658452 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.658381    2579 generic.go:356] "Generic (PLEG): container finished" podID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerID="822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45" exitCode=0
Jul 29 21:44:29.658576 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.658466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" event={"ID":"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7","Type":"ContainerDied","Data":"822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45"}
Jul 29 21:44:29.658857 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.658838    2579 scope.go:117] "RemoveContainer" containerID="822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45"
Jul 29 21:44:29.659673 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.659652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk" event={"ID":"d613b306-f1b4-4080-a7ac-875908b89afb","Type":"ContainerStarted","Data":"49d9092d432dcbf3ed9e28533aa37082ebb5b1222b45ad6fbc90dbbae01ade6a"}
Jul 29 21:44:29.659894 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.659869    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:44:29.659967 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.659942    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:44:29.660007 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:29.659976    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:44:30.670220 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:30.670186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" event={"ID":"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7","Type":"ContainerStarted","Data":"3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524"}
Jul 29 21:44:30.673588 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:30.673563    2579 generic.go:356] "Generic (PLEG): container finished" podID="d613b306-f1b4-4080-a7ac-875908b89afb" containerID="50697ee73394b74be1bb94cddc97225ee5c01f7fc51beeb889af3724fc2eeb86" exitCode=0
Jul 29 21:44:30.673763 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:30.673748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk" event={"ID":"d613b306-f1b4-4080-a7ac-875908b89afb","Type":"ContainerDied","Data":"50697ee73394b74be1bb94cddc97225ee5c01f7fc51beeb889af3724fc2eeb86"}
Jul 29 21:44:30.705475 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:30.705451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:30.705729 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:30.705709    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:44:30.705785 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:30.705754    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 21:44:31.706912 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:31.706872    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:44:31.707260 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:31.706934    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 21:44:31.831133 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:31.831108    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"
Jul 29 21:44:31.983031 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:31.982943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d613b306-f1b4-4080-a7ac-875908b89afb-maas-api-service-ca\") pod \"d613b306-f1b4-4080-a7ac-875908b89afb\" (UID: \"d613b306-f1b4-4080-a7ac-875908b89afb\") "
Jul 29 21:44:31.983263 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:31.983242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d613b306-f1b4-4080-a7ac-875908b89afb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d613b306-f1b4-4080-a7ac-875908b89afb" (UID: "d613b306-f1b4-4080-a7ac-875908b89afb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:44:32.084429 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:32.084404    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d613b306-f1b4-4080-a7ac-875908b89afb-maas-api-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:32.684674 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:32.684643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk" event={"ID":"d613b306-f1b4-4080-a7ac-875908b89afb","Type":"ContainerDied","Data":"49d9092d432dcbf3ed9e28533aa37082ebb5b1222b45ad6fbc90dbbae01ade6a"}
Jul 29 21:44:32.684674 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:32.684670    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"
Jul 29 21:44:32.684674 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:32.684690    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49d9092d432dcbf3ed9e28533aa37082ebb5b1222b45ad6fbc90dbbae01ade6a"
Jul 29 21:44:32.705246 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:32.705221    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:44:32.705373 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:32.705268    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 21:44:33.704749 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:33.704715    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:44:33.705134 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:33.704763    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Jul 29 21:44:34.709635 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:34.709609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:34.710109 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:34.710089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:34.710729 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:34.710715    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:38.131021 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:38.130980    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"]
Jul 29 21:44:38.132764 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:38.132744    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-50783d-hbvtk"]
Jul 29 21:44:39.256731 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:39.256699    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d613b306-f1b4-4080-a7ac-875908b89afb" path="/var/lib/kubelet/pods/d613b306-f1b4-4080-a7ac-875908b89afb/volumes"
Jul 29 21:44:45.128148 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:45.128120    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"]
Jul 29 21:44:45.128570 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:45.128351    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy" containerID="cri-o://3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524" gracePeriod=30
Jul 29 21:44:48.676103 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.676071    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"]
Jul 29 21:44:48.676481 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.676461    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d613b306-f1b4-4080-a7ac-875908b89afb" containerName="revoke-keys"
Jul 29 21:44:48.676481 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.676473    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d613b306-f1b4-4080-a7ac-875908b89afb" containerName="revoke-keys"
Jul 29 21:44:48.676557 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.676537    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d613b306-f1b4-4080-a7ac-875908b89afb" containerName="revoke-keys"
Jul 29 21:44:48.679478 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.679463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"
Jul 29 21:44:48.681786 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.681766    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:44:48.681916 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.681767    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:44:48.687188 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.687167    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"]
Jul 29 21:44:48.706473 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.706445    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b602b90-7315-4a86-9461-537b5e081722-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx\" (UID: \"5b602b90-7315-4a86-9461-537b5e081722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"
Jul 29 21:44:48.807087 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.807066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b602b90-7315-4a86-9461-537b5e081722-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx\" (UID: \"5b602b90-7315-4a86-9461-537b5e081722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"
Jul 29 21:44:48.807655 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.807636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b602b90-7315-4a86-9461-537b5e081722-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx\" (UID: \"5b602b90-7315-4a86-9461-537b5e081722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"
Jul 29 21:44:48.990510 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:48.990436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"
Jul 29 21:44:49.118787 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:49.118755    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"]
Jul 29 21:44:49.120693 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:44:49.120665    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b602b90_7315_4a86_9461_537b5e081722.slice/crio-16321c0b2cf4f84fcbe966100afa50ecdb2438be448e90b4fa10ffcccb5495a9 WatchSource:0}: Error finding container 16321c0b2cf4f84fcbe966100afa50ecdb2438be448e90b4fa10ffcccb5495a9: Status 404 returned error can't find the container with id 16321c0b2cf4f84fcbe966100afa50ecdb2438be448e90b4fa10ffcccb5495a9
Jul 29 21:44:49.711798 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:49.711718    2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:44:49.711798 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:49.711781    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:44:49.766217 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:49.766188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx" event={"ID":"5b602b90-7315-4a86-9461-537b5e081722","Type":"ContainerStarted","Data":"2ff4a0d97ed0abd923ead3554f2115f1ec96c9a9a94202b9a27dff8dab5e5284"}
Jul 29 21:44:49.766379 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:49.766235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx" event={"ID":"5b602b90-7315-4a86-9461-537b5e081722","Type":"ContainerStarted","Data":"16321c0b2cf4f84fcbe966100afa50ecdb2438be448e90b4fa10ffcccb5495a9"}
Jul 29 21:44:49.783391 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:49.783342    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx" podStartSLOduration=1.783328622 podStartE2EDuration="1.783328622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:44:49.781403114 +0000 UTC m=+3439.097892029" watchObservedRunningTime="2026-07-29 21:44:49.783328622 +0000 UTC m=+3439.099817558"
Jul 29 21:44:50.377934 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.377907    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:50.422266 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-data\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422266 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istiod-ca-cert\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422266 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422268    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-certs\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422559 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422329    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9cs8\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-kube-api-access-q9cs8\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422559 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422356    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-token\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422559 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422375    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-socket\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422559 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422417    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-envoy\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422559 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422448    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-podinfo\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422559 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-credential-socket\") pod \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\" (UID: \"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7\") "
Jul 29 21:44:50.422559 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:44:50.422917 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:44:50.422917 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422772    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.422917 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422789    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.422917 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:44:50.423069 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.422944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:44:50.423069 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.423004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:44:50.424713 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.424685    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:44:50.424821 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.424763    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:44:50.424887 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.424871    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-kube-api-access-q9cs8" (OuterVolumeSpecName: "kube-api-access-q9cs8") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "kube-api-access-q9cs8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:44:50.424932 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.424903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" (UID: "2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:44:50.524152 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.524123    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.524152 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.524144    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.524152 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.524154    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.524405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.524163    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.524405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.524171    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.524405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.524179    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9cs8\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-kube-api-access-q9cs8\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.524405 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.524187    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:50.773082 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.772989    2579 generic.go:356] "Generic (PLEG): container finished" podID="5b602b90-7315-4a86-9461-537b5e081722" containerID="2ff4a0d97ed0abd923ead3554f2115f1ec96c9a9a94202b9a27dff8dab5e5284" exitCode=0
Jul 29 21:44:50.773082 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.773038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx" event={"ID":"5b602b90-7315-4a86-9461-537b5e081722","Type":"ContainerDied","Data":"2ff4a0d97ed0abd923ead3554f2115f1ec96c9a9a94202b9a27dff8dab5e5284"}
Jul 29 21:44:50.774545 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.774523    2579 generic.go:356] "Generic (PLEG): container finished" podID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerID="3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524" exitCode=0
Jul 29 21:44:50.774628 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.774565    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"
Jul 29 21:44:50.774628 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.774550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" event={"ID":"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7","Type":"ContainerDied","Data":"3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524"}
Jul 29 21:44:50.774628 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.774616    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc" event={"ID":"2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7","Type":"ContainerDied","Data":"185c4555f2aa2103caea9f7666f34c1127ee7ad07cb6c297031c4bb69e2b5cb8"}
Jul 29 21:44:50.774725 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.774643    2579 scope.go:117] "RemoveContainer" containerID="3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524"
Jul 29 21:44:50.783766 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.783748    2579 scope.go:117] "RemoveContainer" containerID="822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45"
Jul 29 21:44:50.791616 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.791597    2579 scope.go:117] "RemoveContainer" containerID="3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524"
Jul 29 21:44:50.791882 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:44:50.791858    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524\": container with ID starting with 3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524 not found: ID does not exist" containerID="3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524"
Jul 29 21:44:50.791963 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.791887    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524"} err="failed to get container status \"3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524\": rpc error: code = NotFound desc = could not find container \"3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524\": container with ID starting with 3a897ad352b48227482862c0fae9523af7283a58511d4ccf0279b04691dff524 not found: ID does not exist"
Jul 29 21:44:50.791963 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.791904    2579 scope.go:117] "RemoveContainer" containerID="822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45"
Jul 29 21:44:50.792173 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:44:50.792156    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45\": container with ID starting with 822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45 not found: ID does not exist" containerID="822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45"
Jul 29 21:44:50.792214 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.792186    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45"} err="failed to get container status \"822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45\": rpc error: code = NotFound desc = could not find container \"822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45\": container with ID starting with 822f68cd55f2b7177c8efd71efca405a34674d360ac3d6206c97d6abbfa3da45 not found: ID does not exist"
Jul 29 21:44:50.802721 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.802702    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"]
Jul 29 21:44:50.807348 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:50.807325    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-50783d-openshift-default-5bf494f76b-4x7dc"]
Jul 29 21:44:51.255458 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:51.255429    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" path="/var/lib/kubelet/pods/2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7/volumes"
Jul 29 21:44:51.910564 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:51.910535    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"
Jul 29 21:44:51.935196 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:51.935172    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b602b90-7315-4a86-9461-537b5e081722-maas-api-service-ca\") pod \"5b602b90-7315-4a86-9461-537b5e081722\" (UID: \"5b602b90-7315-4a86-9461-537b5e081722\") "
Jul 29 21:44:51.935516 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:51.935496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b602b90-7315-4a86-9461-537b5e081722-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b602b90-7315-4a86-9461-537b5e081722" (UID: "5b602b90-7315-4a86-9461-537b5e081722"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:44:52.035933 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:52.035859    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b602b90-7315-4a86-9461-537b5e081722-maas-api-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:44:52.785984 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:52.785958    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"
Jul 29 21:44:52.786151 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:52.785954    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx" event={"ID":"5b602b90-7315-4a86-9461-537b5e081722","Type":"ContainerDied","Data":"16321c0b2cf4f84fcbe966100afa50ecdb2438be448e90b4fa10ffcccb5495a9"}
Jul 29 21:44:52.786151 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:44:52.786086    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16321c0b2cf4f84fcbe966100afa50ecdb2438be448e90b4fa10ffcccb5495a9"
Jul 29 21:45:00.135923 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.135891    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm"]
Jul 29 21:45:00.136306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136238    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy"
Jul 29 21:45:00.136306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136250    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy"
Jul 29 21:45:00.136306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136266    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy"
Jul 29 21:45:00.136306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136274    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy"
Jul 29 21:45:00.136306 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136303    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b602b90-7315-4a86-9461-537b5e081722" containerName="revoke-keys"
Jul 29 21:45:00.136473 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136311    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b602b90-7315-4a86-9461-537b5e081722" containerName="revoke-keys"
Jul 29 21:45:00.136473 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136395    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b602b90-7315-4a86-9461-537b5e081722" containerName="revoke-keys"
Jul 29 21:45:00.136473 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.136405    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy"
Jul 29 21:45:00.139882 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.139860    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm"
Jul 29 21:45:00.140831 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.140806    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"]
Jul 29 21:45:00.141214 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.141196    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff57fac-cd7a-45a7-9ef5-f98448c1a5c7" containerName="istio-proxy"
Jul 29 21:45:00.142202 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.142182    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:45:00.144161 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.144141    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"]
Jul 29 21:45:00.144311 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.144268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"
Jul 29 21:45:00.147027 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.147009    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"]
Jul 29 21:45:00.147133 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.147073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"
Jul 29 21:45:00.158002 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.157923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm"
Jul 29 21:45:00.168364 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.168345    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"]
Jul 29 21:45:00.170048 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.169936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"
Jul 29 21:45:00.172451 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.172431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm"]
Jul 29 21:45:00.172562 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.172465    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"]
Jul 29 21:45:00.172562 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.172482    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"]
Jul 29 21:45:00.172562 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.172500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"]
Jul 29 21:45:00.172562 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.172564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"
Jul 29 21:45:00.173158 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.173142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"
Jul 29 21:45:00.190083 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.190020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"]
Jul 29 21:45:00.211530 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.211502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"
Jul 29 21:45:00.219976 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.219941    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"
Jul 29 21:45:00.258979 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.258955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"
Jul 29 21:45:00.365371 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.365278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm"]
Jul 29 21:45:00.392324 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.392215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"]
Jul 29 21:45:00.398549 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:45:00.398476    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53b64b1_83bb_40b2_8c39_eb9dad5da64c.slice/crio-251dd040b718ece62e8b113054ce989d0ab0c4c727d825dfbc45af0a2d9d64a8 WatchSource:0}: Error finding container 251dd040b718ece62e8b113054ce989d0ab0c4c727d825dfbc45af0a2d9d64a8: Status 404 returned error can't find the container with id 251dd040b718ece62e8b113054ce989d0ab0c4c727d825dfbc45af0a2d9d64a8
Jul 29 21:45:00.446578 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.446545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"]
Jul 29 21:45:00.448589 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:45:00.448565    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode036ebb6_3e39_421d_89a0_67c16f2e7d90.slice/crio-ffd2621fdde60286001609c5065653191d08ba4f9faf8e59c6921926d7a8c876 WatchSource:0}: Error finding container ffd2621fdde60286001609c5065653191d08ba4f9faf8e59c6921926d7a8c876: Status 404 returned error can't find the container with id ffd2621fdde60286001609c5065653191d08ba4f9faf8e59c6921926d7a8c876
Jul 29 21:45:00.469073 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.469042    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"]
Jul 29 21:45:00.470540 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:45:00.470520    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a45f78_3bc0_4e14_8df7_df6aa4e4b565.slice/crio-afaeba1f17f5b4d6626234cd0977765056c94a0b6d7511ca300c6744cac2bd6f WatchSource:0}: Error finding container afaeba1f17f5b4d6626234cd0977765056c94a0b6d7511ca300c6744cac2bd6f: Status 404 returned error can't find the container with id afaeba1f17f5b4d6626234cd0977765056c94a0b6d7511ca300c6744cac2bd6f
Jul 29 21:45:00.625793 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.625770    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"]
Jul 29 21:45:00.627019 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:45:00.626999    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbbbf45a_a374_4039_9f5b_84810c7777db.slice/crio-1313de0d214069f6a4d4b0fb178958a273b36a109f7b1d61c44b3c7fff961917 WatchSource:0}: Error finding container 1313de0d214069f6a4d4b0fb178958a273b36a109f7b1d61c44b3c7fff961917: Status 404 returned error can't find the container with id 1313de0d214069f6a4d4b0fb178958a273b36a109f7b1d61c44b3c7fff961917
Jul 29 21:45:00.824526 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.824498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2" event={"ID":"e036ebb6-3e39-421d-89a0-67c16f2e7d90","Type":"ContainerStarted","Data":"8c584c319202724cfb185bbd0cae606905221b0d09caa5e6b86b15e25e1b6c9d"}
Jul 29 21:45:00.824647 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.824556    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2" event={"ID":"e036ebb6-3e39-421d-89a0-67c16f2e7d90","Type":"ContainerStarted","Data":"ffd2621fdde60286001609c5065653191d08ba4f9faf8e59c6921926d7a8c876"}
Jul 29 21:45:00.825966 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.825945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c" event={"ID":"f53b64b1-83bb-40b2-8c39-eb9dad5da64c","Type":"ContainerStarted","Data":"398b57a9afefc67d6fd71fd5330de835ff69b67750ba58cf8bae0763c9eb136a"}
Jul 29 21:45:00.826058 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.825989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c" event={"ID":"f53b64b1-83bb-40b2-8c39-eb9dad5da64c","Type":"ContainerStarted","Data":"251dd040b718ece62e8b113054ce989d0ab0c4c727d825dfbc45af0a2d9d64a8"}
Jul 29 21:45:00.827339 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.827309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm" event={"ID":"fd92bc73-78ba-4c88-b6f2-01232f190ce7","Type":"ContainerStarted","Data":"f7ea558570e826cf076bda313ef0cd108f9f7934f0a083e6213bc6251db3a3d3"}
Jul 29 21:45:00.827446 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.827367    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm" event={"ID":"fd92bc73-78ba-4c88-b6f2-01232f190ce7","Type":"ContainerStarted","Data":"e6a775c00cd325817cdd952a29c503b3be027dc3ef1a5f55c40de730bd5dcf16"}
Jul 29 21:45:00.828482 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.828462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx" event={"ID":"30a45f78-3bc0-4e14-8df7-df6aa4e4b565","Type":"ContainerStarted","Data":"afaeba1f17f5b4d6626234cd0977765056c94a0b6d7511ca300c6744cac2bd6f"}
Jul 29 21:45:00.829459 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.829441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp" event={"ID":"bbbbf45a-a374-4039-9f5b-84810c7777db","Type":"ContainerStarted","Data":"1313de0d214069f6a4d4b0fb178958a273b36a109f7b1d61c44b3c7fff961917"}
Jul 29 21:45:00.842813 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.842770    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2" podStartSLOduration=0.842759107 podStartE2EDuration="842.759107ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:45:00.84020071 +0000 UTC m=+3450.156689618" watchObservedRunningTime="2026-07-29 21:45:00.842759107 +0000 UTC m=+3450.159248021"
Jul 29 21:45:00.855393 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.855283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm" podStartSLOduration=0.855267747 podStartE2EDuration="855.267747ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:45:00.854510638 +0000 UTC m=+3450.170999553" watchObservedRunningTime="2026-07-29 21:45:00.855267747 +0000 UTC m=+3450.171756661"
Jul 29 21:45:00.868714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:00.868682    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c" podStartSLOduration=0.868672502 podStartE2EDuration="868.672502ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:45:00.867428738 +0000 UTC m=+3450.183917654" watchObservedRunningTime="2026-07-29 21:45:00.868672502 +0000 UTC m=+3450.185161416"
Jul 29 21:45:01.834944 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.834911    2579 generic.go:356] "Generic (PLEG): container finished" podID="e036ebb6-3e39-421d-89a0-67c16f2e7d90" containerID="8c584c319202724cfb185bbd0cae606905221b0d09caa5e6b86b15e25e1b6c9d" exitCode=0
Jul 29 21:45:01.835391 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.834986    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2" event={"ID":"e036ebb6-3e39-421d-89a0-67c16f2e7d90","Type":"ContainerDied","Data":"8c584c319202724cfb185bbd0cae606905221b0d09caa5e6b86b15e25e1b6c9d"}
Jul 29 21:45:01.836411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.836388    2579 generic.go:356] "Generic (PLEG): container finished" podID="f53b64b1-83bb-40b2-8c39-eb9dad5da64c" containerID="398b57a9afefc67d6fd71fd5330de835ff69b67750ba58cf8bae0763c9eb136a" exitCode=0
Jul 29 21:45:01.836499 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.836464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c" event={"ID":"f53b64b1-83bb-40b2-8c39-eb9dad5da64c","Type":"ContainerDied","Data":"398b57a9afefc67d6fd71fd5330de835ff69b67750ba58cf8bae0763c9eb136a"}
Jul 29 21:45:01.837856 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.837838    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd92bc73-78ba-4c88-b6f2-01232f190ce7" containerID="f7ea558570e826cf076bda313ef0cd108f9f7934f0a083e6213bc6251db3a3d3" exitCode=0
Jul 29 21:45:01.837947 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.837862    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm" event={"ID":"fd92bc73-78ba-4c88-b6f2-01232f190ce7","Type":"ContainerDied","Data":"f7ea558570e826cf076bda313ef0cd108f9f7934f0a083e6213bc6251db3a3d3"}
Jul 29 21:45:01.839203 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.839184    2579 generic.go:356] "Generic (PLEG): container finished" podID="30a45f78-3bc0-4e14-8df7-df6aa4e4b565" containerID="aa43d620854dfb2028819b260a43bed9dd33417a096d8164587d5807e02085c2" exitCode=0
Jul 29 21:45:01.839312 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.839247    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx" event={"ID":"30a45f78-3bc0-4e14-8df7-df6aa4e4b565","Type":"ContainerDied","Data":"aa43d620854dfb2028819b260a43bed9dd33417a096d8164587d5807e02085c2"}
Jul 29 21:45:01.840772 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.840752    2579 generic.go:356] "Generic (PLEG): container finished" podID="bbbbf45a-a374-4039-9f5b-84810c7777db" containerID="a1e63b28708b81edc43f7d2ebb4fed9ef7a7a1e1b3a39ae7d613f6a97f003e75" exitCode=0
Jul 29 21:45:01.840845 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:01.840779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp" event={"ID":"bbbbf45a-a374-4039-9f5b-84810c7777db","Type":"ContainerDied","Data":"a1e63b28708b81edc43f7d2ebb4fed9ef7a7a1e1b3a39ae7d613f6a97f003e75"}
Jul 29 21:45:02.042034 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:02.041955    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"]
Jul 29 21:45:02.300953 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:02.300873    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"]
Jul 29 21:45:02.305167 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:02.305139    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f2f2fe-p48nx"]
Jul 29 21:45:02.994686 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:02.994664    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"
Jul 29 21:45:03.029872 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.029841    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"
Jul 29 21:45:03.033515 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.033484    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"
Jul 29 21:45:03.062328 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.062264    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm"
Jul 29 21:45:03.068999 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.068975    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"
Jul 29 21:45:03.257940 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.257911    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b602b90-7315-4a86-9461-537b5e081722" path="/var/lib/kubelet/pods/5b602b90-7315-4a86-9461-537b5e081722/volumes"
Jul 29 21:45:03.852411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.852384    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm"
Jul 29 21:45:03.852411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.852391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-xqsvm" event={"ID":"fd92bc73-78ba-4c88-b6f2-01232f190ce7","Type":"ContainerDied","Data":"e6a775c00cd325817cdd952a29c503b3be027dc3ef1a5f55c40de730bd5dcf16"}
Jul 29 21:45:03.852627 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.852431    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a775c00cd325817cdd952a29c503b3be027dc3ef1a5f55c40de730bd5dcf16"
Jul 29 21:45:03.853706 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.853688    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"
Jul 29 21:45:03.853828 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.853702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx" event={"ID":"30a45f78-3bc0-4e14-8df7-df6aa4e4b565","Type":"ContainerDied","Data":"afaeba1f17f5b4d6626234cd0977765056c94a0b6d7511ca300c6744cac2bd6f"}
Jul 29 21:45:03.853828 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.853741    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afaeba1f17f5b4d6626234cd0977765056c94a0b6d7511ca300c6744cac2bd6f"
Jul 29 21:45:03.855042 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.855019    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"
Jul 29 21:45:03.855141 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.855067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp" event={"ID":"bbbbf45a-a374-4039-9f5b-84810c7777db","Type":"ContainerDied","Data":"1313de0d214069f6a4d4b0fb178958a273b36a109f7b1d61c44b3c7fff961917"}
Jul 29 21:45:03.855141 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.855102    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1313de0d214069f6a4d4b0fb178958a273b36a109f7b1d61c44b3c7fff961917"
Jul 29 21:45:03.856512 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.856494    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"
Jul 29 21:45:03.856625 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.856500    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2" event={"ID":"e036ebb6-3e39-421d-89a0-67c16f2e7d90","Type":"ContainerDied","Data":"ffd2621fdde60286001609c5065653191d08ba4f9faf8e59c6921926d7a8c876"}
Jul 29 21:45:03.856688 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.856645    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd2621fdde60286001609c5065653191d08ba4f9faf8e59c6921926d7a8c876"
Jul 29 21:45:03.857892 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.857874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c" event={"ID":"f53b64b1-83bb-40b2-8c39-eb9dad5da64c","Type":"ContainerDied","Data":"251dd040b718ece62e8b113054ce989d0ab0c4c727d825dfbc45af0a2d9d64a8"}
Jul 29 21:45:03.857993 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.857902    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"
Jul 29 21:45:03.857993 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.857904    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="251dd040b718ece62e8b113054ce989d0ab0c4c727d825dfbc45af0a2d9d64a8"
Jul 29 21:45:03.885706 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.885685    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"]
Jul 29 21:45:03.889908 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:03.889886    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f2f2fe-29756025-t48z2"]
Jul 29 21:45:05.255161 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:05.255129    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e036ebb6-3e39-421d-89a0-67c16f2e7d90" path="/var/lib/kubelet/pods/e036ebb6-3e39-421d-89a0-67c16f2e7d90/volumes"
Jul 29 21:45:07.610173 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:07.610141    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"]
Jul 29 21:45:22.628431 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.628388    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4" podUID="3c65614d-7089-4ba0-9547-6fe9a3823c03" containerName="main" containerID="cri-o://61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd" gracePeriod=45
Jul 29 21:45:22.871398 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.871371    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:45:22.955543 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.955460    2579 generic.go:356] "Generic (PLEG): container finished" podID="3c65614d-7089-4ba0-9547-6fe9a3823c03" containerID="61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd" exitCode=0
Jul 29 21:45:22.955543 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.955504    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"
Jul 29 21:45:22.955739 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.955544    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4" event={"ID":"3c65614d-7089-4ba0-9547-6fe9a3823c03","Type":"ContainerDied","Data":"61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd"}
Jul 29 21:45:22.955739 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.955585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4" event={"ID":"3c65614d-7089-4ba0-9547-6fe9a3823c03","Type":"ContainerDied","Data":"1c10e58d8f9e660510e4873fd279b72e4cea15b32824673ce7387bc399f359e4"}
Jul 29 21:45:22.955739 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.955607    2579 scope.go:117] "RemoveContainer" containerID="61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd"
Jul 29 21:45:22.963647 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.963630    2579 scope.go:117] "RemoveContainer" containerID="61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd"
Jul 29 21:45:22.963912 ip-10-0-136-64 kubenswrapper[2579]: E0729 21:45:22.963894    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd\": container with ID starting with 61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd not found: ID does not exist" containerID="61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd"
Jul 29 21:45:22.963963 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.963921    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd"} err="failed to get container status \"61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd\": rpc error: code = NotFound desc = could not find container \"61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd\": container with ID starting with 61228bed0c1f3b462ecd6fd9e354b6bf08764580c9ab0d44fbce880d75c3d6dd not found: ID does not exist"
Jul 29 21:45:22.977211 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977188    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-home\") pod \"3c65614d-7089-4ba0-9547-6fe9a3823c03\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") "
Jul 29 21:45:22.977343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977255    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-tmp-dir\") pod \"3c65614d-7089-4ba0-9547-6fe9a3823c03\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") "
Jul 29 21:45:22.977343 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr52c\" (UniqueName: \"kubernetes.io/projected/3c65614d-7089-4ba0-9547-6fe9a3823c03-kube-api-access-wr52c\") pod \"3c65614d-7089-4ba0-9547-6fe9a3823c03\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") "
Jul 29 21:45:22.977456 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977347    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-dshm\") pod \"3c65614d-7089-4ba0-9547-6fe9a3823c03\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") "
Jul 29 21:45:22.977456 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c65614d-7089-4ba0-9547-6fe9a3823c03-tls-certs\") pod \"3c65614d-7089-4ba0-9547-6fe9a3823c03\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") "
Jul 29 21:45:22.977456 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977422    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-model-cache\") pod \"3c65614d-7089-4ba0-9547-6fe9a3823c03\" (UID: \"3c65614d-7089-4ba0-9547-6fe9a3823c03\") "
Jul 29 21:45:22.977456 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977447    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c65614d-7089-4ba0-9547-6fe9a3823c03" (UID: "3c65614d-7089-4ba0-9547-6fe9a3823c03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:22.977649 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-home" (OuterVolumeSpecName: "home") pod "3c65614d-7089-4ba0-9547-6fe9a3823c03" (UID: "3c65614d-7089-4ba0-9547-6fe9a3823c03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:22.977701 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977670    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-home\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:45:22.977701 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c65614d-7089-4ba0-9547-6fe9a3823c03" (UID: "3c65614d-7089-4ba0-9547-6fe9a3823c03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:22.977778 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.977686    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-tmp-dir\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:45:22.979355 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.979335    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-dshm" (OuterVolumeSpecName: "dshm") pod "3c65614d-7089-4ba0-9547-6fe9a3823c03" (UID: "3c65614d-7089-4ba0-9547-6fe9a3823c03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:45:22.979425 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.979412    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c65614d-7089-4ba0-9547-6fe9a3823c03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c65614d-7089-4ba0-9547-6fe9a3823c03" (UID: "3c65614d-7089-4ba0-9547-6fe9a3823c03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:45:22.979481 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:22.979463    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c65614d-7089-4ba0-9547-6fe9a3823c03-kube-api-access-wr52c" (OuterVolumeSpecName: "kube-api-access-wr52c") pod "3c65614d-7089-4ba0-9547-6fe9a3823c03" (UID: "3c65614d-7089-4ba0-9547-6fe9a3823c03"). InnerVolumeSpecName "kube-api-access-wr52c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:45:23.078307 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:23.078258    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c65614d-7089-4ba0-9547-6fe9a3823c03-tls-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:45:23.078307 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:23.078281    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-model-cache\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:45:23.078307 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:23.078310    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr52c\" (UniqueName: \"kubernetes.io/projected/3c65614d-7089-4ba0-9547-6fe9a3823c03-kube-api-access-wr52c\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:45:23.078523 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:23.078319    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c65614d-7089-4ba0-9547-6fe9a3823c03-dshm\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:45:23.279776 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:23.279749    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"]
Jul 29 21:45:23.281412 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:23.281393    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-f2f2fe/ipp-route-f2f2fe-kserve-788f56447f-xxxg4"]
Jul 29 21:45:25.255893 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:25.255862    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c65614d-7089-4ba0-9547-6fe9a3823c03" path="/var/lib/kubelet/pods/3c65614d-7089-4ba0-9547-6fe9a3823c03/volumes"
Jul 29 21:45:51.824996 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.824963    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"]
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825322    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f53b64b1-83bb-40b2-8c39-eb9dad5da64c" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825334    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f53b64b1-83bb-40b2-8c39-eb9dad5da64c" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825347    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd92bc73-78ba-4c88-b6f2-01232f190ce7" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825354    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd92bc73-78ba-4c88-b6f2-01232f190ce7" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825363    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbbf45a-a374-4039-9f5b-84810c7777db" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825367    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbbf45a-a374-4039-9f5b-84810c7777db" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825376    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c65614d-7089-4ba0-9547-6fe9a3823c03" containerName="main"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825381    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c65614d-7089-4ba0-9547-6fe9a3823c03" containerName="main"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825388    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30a45f78-3bc0-4e14-8df7-df6aa4e4b565" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825394    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="30a45f78-3bc0-4e14-8df7-df6aa4e4b565" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825408    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e036ebb6-3e39-421d-89a0-67c16f2e7d90" containerName="cleanup"
Jul 29 21:45:51.825410 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825414    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e036ebb6-3e39-421d-89a0-67c16f2e7d90" containerName="cleanup"
Jul 29 21:45:51.825807 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825468    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbbf45a-a374-4039-9f5b-84810c7777db" containerName="cleanup"
Jul 29 21:45:51.825807 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825477    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="30a45f78-3bc0-4e14-8df7-df6aa4e4b565" containerName="cleanup"
Jul 29 21:45:51.825807 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825483    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e036ebb6-3e39-421d-89a0-67c16f2e7d90" containerName="cleanup"
Jul 29 21:45:51.825807 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825488    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f53b64b1-83bb-40b2-8c39-eb9dad5da64c" containerName="cleanup"
Jul 29 21:45:51.825807 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825494    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd92bc73-78ba-4c88-b6f2-01232f190ce7" containerName="cleanup"
Jul 29 21:45:51.825807 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.825498    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c65614d-7089-4ba0-9547-6fe9a3823c03" containerName="main"
Jul 29 21:45:51.828421 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.828404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"
Jul 29 21:45:51.830563 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.830545    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:45:51.830684 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.830547    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:45:51.835731 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.835709    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"]
Jul 29 21:45:51.997936 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:51.997899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad344d3-4445-495a-8ed8-28a066437fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52\" (UID: \"2ad344d3-4445-495a-8ed8-28a066437fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"
Jul 29 21:45:52.099092 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:52.099064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad344d3-4445-495a-8ed8-28a066437fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52\" (UID: \"2ad344d3-4445-495a-8ed8-28a066437fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"
Jul 29 21:45:52.099651 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:52.099634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad344d3-4445-495a-8ed8-28a066437fe6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52\" (UID: \"2ad344d3-4445-495a-8ed8-28a066437fe6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"
Jul 29 21:45:52.139128 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:52.139105    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"
Jul 29 21:45:52.265715 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:52.265694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"]
Jul 29 21:45:52.267947 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:45:52.267919    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad344d3_4445_495a_8ed8_28a066437fe6.slice/crio-7c1de5515b383100f8d356ca77da8618e45489217c4bfc9a5521b7d9cb64d45a WatchSource:0}: Error finding container 7c1de5515b383100f8d356ca77da8618e45489217c4bfc9a5521b7d9cb64d45a: Status 404 returned error can't find the container with id 7c1de5515b383100f8d356ca77da8618e45489217c4bfc9a5521b7d9cb64d45a
Jul 29 21:45:53.104398 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:53.104361    2579 generic.go:356] "Generic (PLEG): container finished" podID="2ad344d3-4445-495a-8ed8-28a066437fe6" containerID="e906f3047053282102e989df6ba07c1dfe80902daf1dd922a326f1bc68a74676" exitCode=0
Jul 29 21:45:53.104770 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:53.104444    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52" event={"ID":"2ad344d3-4445-495a-8ed8-28a066437fe6","Type":"ContainerDied","Data":"e906f3047053282102e989df6ba07c1dfe80902daf1dd922a326f1bc68a74676"}
Jul 29 21:45:53.104770 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:53.104516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52" event={"ID":"2ad344d3-4445-495a-8ed8-28a066437fe6","Type":"ContainerStarted","Data":"7c1de5515b383100f8d356ca77da8618e45489217c4bfc9a5521b7d9cb64d45a"}
Jul 29 21:45:54.233713 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:54.233691    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"
Jul 29 21:45:54.417578 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:54.417504    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad344d3-4445-495a-8ed8-28a066437fe6-maas-api-service-ca\") pod \"2ad344d3-4445-495a-8ed8-28a066437fe6\" (UID: \"2ad344d3-4445-495a-8ed8-28a066437fe6\") "
Jul 29 21:45:54.417864 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:54.417840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ad344d3-4445-495a-8ed8-28a066437fe6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ad344d3-4445-495a-8ed8-28a066437fe6" (UID: "2ad344d3-4445-495a-8ed8-28a066437fe6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:45:54.518528 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:54.518501    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ad344d3-4445-495a-8ed8-28a066437fe6-maas-api-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:45:55.115039 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:55.115013    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"
Jul 29 21:45:55.115039 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:55.115031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52" event={"ID":"2ad344d3-4445-495a-8ed8-28a066437fe6","Type":"ContainerDied","Data":"7c1de5515b383100f8d356ca77da8618e45489217c4bfc9a5521b7d9cb64d45a"}
Jul 29 21:45:55.115240 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:55.115076    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c1de5515b383100f8d356ca77da8618e45489217c4bfc9a5521b7d9cb64d45a"
Jul 29 21:45:58.581538 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:58.581506    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"]
Jul 29 21:45:58.584934 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:58.584912    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-00731e-29756025-95qcp"]
Jul 29 21:45:58.823404 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:58.823364    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"]
Jul 29 21:45:58.827000 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:58.826976    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-00731e-wdv52"]
Jul 29 21:45:59.255395 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:59.255358    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ad344d3-4445-495a-8ed8-28a066437fe6" path="/var/lib/kubelet/pods/2ad344d3-4445-495a-8ed8-28a066437fe6/volumes"
Jul 29 21:45:59.255745 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:45:59.255726    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbbbf45a-a374-4039-9f5b-84810c7777db" path="/var/lib/kubelet/pods/bbbbf45a-a374-4039-9f5b-84810c7777db/volumes"
Jul 29 21:46:06.034375 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:06.034346    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"]
Jul 29 21:46:06.034831 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:06.034547    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy" containerID="cri-o://c778f9fbd7f22da4f4aa9d501e27e0ba5c4831da2820ed449c8827bb340e3b6d" gracePeriod=30
Jul 29 21:46:10.836797 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:10.836761    2579 patch_prober.go:28] interesting pod/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:46:10.837160 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:10.836841    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:46:11.200256 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.200230    2579 generic.go:356] "Generic (PLEG): container finished" podID="7882195f-a528-4807-a3d6-15c6e92f2578" containerID="c778f9fbd7f22da4f4aa9d501e27e0ba5c4831da2820ed449c8827bb340e3b6d" exitCode=0
Jul 29 21:46:11.200406 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.200320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" event={"ID":"7882195f-a528-4807-a3d6-15c6e92f2578","Type":"ContainerDied","Data":"c778f9fbd7f22da4f4aa9d501e27e0ba5c4831da2820ed449c8827bb340e3b6d"}
Jul 29 21:46:11.200406 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.200381    2579 scope.go:117] "RemoveContainer" containerID="944d22c0af337d6b437b12da83b6dad94d68dea8c30b3c4c092f4157c1e7d06a"
Jul 29 21:46:11.270393 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.270369    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:46:11.343363 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882195f-a528-4807-a3d6-15c6e92f2578-istio-podinfo\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343363 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-credential-socket\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343555 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882195f-a528-4807-a3d6-15c6e92f2578-istiod-ca-cert\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343555 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-data\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343555 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-istio-token\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343555 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343510    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgfh9\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-kube-api-access-bgfh9\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343555 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343538    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-certs\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343855 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-socket\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343855 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343602    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-envoy\") pod \"7882195f-a528-4807-a3d6-15c6e92f2578\" (UID: \"7882195f-a528-4807-a3d6-15c6e92f2578\") "
Jul 29 21:46:11.343855 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:46:11.343855 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:46:11.343855 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343842    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-credential-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.343855 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.343844    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:46:11.344142 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.344016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7882195f-a528-4807-a3d6-15c6e92f2578-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:46:11.344356 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.344334    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-data" (OuterVolumeSpecName: "istio-data") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:46:11.345517 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.345496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7882195f-a528-4807-a3d6-15c6e92f2578-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:46:11.345650 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.345630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:46:11.345954 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.345932    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-istio-token" (OuterVolumeSpecName: "istio-token") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:46:11.346007 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.345943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-kube-api-access-bgfh9" (OuterVolumeSpecName: "kube-api-access-bgfh9") pod "7882195f-a528-4807-a3d6-15c6e92f2578" (UID: "7882195f-a528-4807-a3d6-15c6e92f2578"). InnerVolumeSpecName "kube-api-access-bgfh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:46:11.445193 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445130    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-socket\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.445193 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445155    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-envoy\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.445193 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445168    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7882195f-a528-4807-a3d6-15c6e92f2578-istio-podinfo\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.445193 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445179    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7882195f-a528-4807-a3d6-15c6e92f2578-istiod-ca-cert\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.445193 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445190    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-istio-data\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.445472 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445201    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-istio-token\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.445472 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445211    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgfh9\" (UniqueName: \"kubernetes.io/projected/7882195f-a528-4807-a3d6-15c6e92f2578-kube-api-access-bgfh9\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:11.445472 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:11.445223    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7882195f-a528-4807-a3d6-15c6e92f2578-workload-certs\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:12.206649 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:12.206622    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"
Jul 29 21:46:12.207084 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:12.206668    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp" event={"ID":"7882195f-a528-4807-a3d6-15c6e92f2578","Type":"ContainerDied","Data":"b5e44582951c9a5f3809985f18b4308b4383d27f634434d6e35dcf1344897b1d"}
Jul 29 21:46:12.207084 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:12.206724    2579 scope.go:117] "RemoveContainer" containerID="c778f9fbd7f22da4f4aa9d501e27e0ba5c4831da2820ed449c8827bb340e3b6d"
Jul 29 21:46:12.213825 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:12.213803    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"]
Jul 29 21:46:12.216459 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:12.216440    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-eb3087-29756025-hgg8c"]
Jul 29 21:46:12.233284 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:12.233260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"]
Jul 29 21:46:12.238710 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:12.238689    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-00731e-openshift-default-6c9dd69568-99kjp"]
Jul 29 21:46:13.255466 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:13.255430    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" path="/var/lib/kubelet/pods/7882195f-a528-4807-a3d6-15c6e92f2578/volumes"
Jul 29 21:46:13.255832 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:13.255728    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f53b64b1-83bb-40b2-8c39-eb9dad5da64c" path="/var/lib/kubelet/pods/f53b64b1-83bb-40b2-8c39-eb9dad5da64c/volumes"
Jul 29 21:46:20.229468 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229435    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"]
Jul 29 21:46:20.229847 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229741    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy"
Jul 29 21:46:20.229847 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229751    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy"
Jul 29 21:46:20.229847 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229771    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ad344d3-4445-495a-8ed8-28a066437fe6" containerName="revoke-keys"
Jul 29 21:46:20.229847 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229776    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ad344d3-4445-495a-8ed8-28a066437fe6" containerName="revoke-keys"
Jul 29 21:46:20.229847 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229833    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy"
Jul 29 21:46:20.229847 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229842    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ad344d3-4445-495a-8ed8-28a066437fe6" containerName="revoke-keys"
Jul 29 21:46:20.229847 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.229850    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy"
Jul 29 21:46:20.232606 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.232588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"
Jul 29 21:46:20.239014 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.238980    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:46:20.239014 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.238992    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zp8cj\""
Jul 29 21:46:20.244695 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.244676    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"]
Jul 29 21:46:20.305714 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.305685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f123d55-3da5-48f2-a4ad-a76fd3286658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-46bb38-762fl\" (UID: \"1f123d55-3da5-48f2-a4ad-a76fd3286658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"
Jul 29 21:46:20.406842 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.406815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f123d55-3da5-48f2-a4ad-a76fd3286658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-46bb38-762fl\" (UID: \"1f123d55-3da5-48f2-a4ad-a76fd3286658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"
Jul 29 21:46:20.407359 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.407344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f123d55-3da5-48f2-a4ad-a76fd3286658-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-46bb38-762fl\" (UID: \"1f123d55-3da5-48f2-a4ad-a76fd3286658\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"
Jul 29 21:46:20.549725 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.549655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"
Jul 29 21:46:20.674958 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:20.674935    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"]
Jul 29 21:46:20.677009 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:46:20.676978    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f123d55_3da5_48f2_a4ad_a76fd3286658.slice/crio-41992a154e470fa38efaf56e4fa1ffb3295a74f18df10cfa5105504ae7f09754 WatchSource:0}: Error finding container 41992a154e470fa38efaf56e4fa1ffb3295a74f18df10cfa5105504ae7f09754: Status 404 returned error can't find the container with id 41992a154e470fa38efaf56e4fa1ffb3295a74f18df10cfa5105504ae7f09754
Jul 29 21:46:21.257064 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:21.256977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl" event={"ID":"1f123d55-3da5-48f2-a4ad-a76fd3286658","Type":"ContainerStarted","Data":"20f1fd64a1e6970e63f1e6f62e0373d8aa5b186368a19fc3369e3ffa3b3c7c8f"}
Jul 29 21:46:21.257064 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:21.257022    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl" event={"ID":"1f123d55-3da5-48f2-a4ad-a76fd3286658","Type":"ContainerStarted","Data":"41992a154e470fa38efaf56e4fa1ffb3295a74f18df10cfa5105504ae7f09754"}
Jul 29 21:46:21.272909 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:21.272859    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl" podStartSLOduration=1.272840201 podStartE2EDuration="1.272840201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:46:21.271627548 +0000 UTC m=+3530.588116461" watchObservedRunningTime="2026-07-29 21:46:21.272840201 +0000 UTC m=+3530.589329116"
Jul 29 21:46:22.263147 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:22.263116    2579 generic.go:356] "Generic (PLEG): container finished" podID="1f123d55-3da5-48f2-a4ad-a76fd3286658" containerID="20f1fd64a1e6970e63f1e6f62e0373d8aa5b186368a19fc3369e3ffa3b3c7c8f" exitCode=0
Jul 29 21:46:22.263548 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:22.263196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl" event={"ID":"1f123d55-3da5-48f2-a4ad-a76fd3286658","Type":"ContainerDied","Data":"20f1fd64a1e6970e63f1e6f62e0373d8aa5b186368a19fc3369e3ffa3b3c7c8f"}
Jul 29 21:46:23.390147 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:23.390127    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"
Jul 29 21:46:23.428415 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:23.428393    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f123d55-3da5-48f2-a4ad-a76fd3286658-maas-api-service-ca\") pod \"1f123d55-3da5-48f2-a4ad-a76fd3286658\" (UID: \"1f123d55-3da5-48f2-a4ad-a76fd3286658\") "
Jul 29 21:46:23.428733 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:23.428714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f123d55-3da5-48f2-a4ad-a76fd3286658-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f123d55-3da5-48f2-a4ad-a76fd3286658" (UID: "1f123d55-3da5-48f2-a4ad-a76fd3286658"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:46:23.529112 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:23.529061    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f123d55-3da5-48f2-a4ad-a76fd3286658-maas-api-service-ca\") on node \"ip-10-0-136-64.ec2.internal\" DevicePath \"\""
Jul 29 21:46:24.274411 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:24.274380    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"
Jul 29 21:46:24.274577 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:24.274387    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl" event={"ID":"1f123d55-3da5-48f2-a4ad-a76fd3286658","Type":"ContainerDied","Data":"41992a154e470fa38efaf56e4fa1ffb3295a74f18df10cfa5105504ae7f09754"}
Jul 29 21:46:24.274577 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:24.274488    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41992a154e470fa38efaf56e4fa1ffb3295a74f18df10cfa5105504ae7f09754"
Jul 29 21:46:25.555393 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:25.555334    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"]
Jul 29 21:46:25.557303 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:25.557264    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-46bb38-29756025-8sddx"]
Jul 29 21:46:25.800251 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:25.800212    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"]
Jul 29 21:46:25.804528 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:25.804502    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-46bb38-762fl"]
Jul 29 21:46:27.256033 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:27.255999    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f123d55-3da5-48f2-a4ad-a76fd3286658" path="/var/lib/kubelet/pods/1f123d55-3da5-48f2-a4ad-a76fd3286658/volumes"
Jul 29 21:46:27.256414 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:27.256247    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30a45f78-3bc0-4e14-8df7-df6aa4e4b565" path="/var/lib/kubelet/pods/30a45f78-3bc0-4e14-8df7-df6aa4e4b565/volumes"
Jul 29 21:46:39.690016 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:39.689990    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-n9dbs_3ca56f15-86bc-4c57-9025-b0dbfde96e35/copy-manifests/0.log"
Jul 29 21:46:39.708104 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:39.708075    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-n9dbs_3ca56f15-86bc-4c57-9025-b0dbfde96e35/manager/0.log"
Jul 29 21:46:40.486165 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:40.486136    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756025-xqsvm_fd92bc73-78ba-4c88-b6f2-01232f190ce7/cleanup/0.log"
Jul 29 21:46:42.264887 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:42.264856    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7rqvx_048aaed8-8b03-40e2-ab2a-7fa6f2e62f99/kuadrant-console-plugin/0.log"
Jul 29 21:46:42.780651 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:42.780625    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lkksg_4de8b85f-818f-4e3e-8bb1-fb19b637b0d6/manager/0.log"
Jul 29 21:46:43.494820 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:43.494794    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-wjm28_643b3181-bfa7-4674-9428-a23fba762430/kube-auth-proxy/0.log"
Jul 29 21:46:44.454122 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:44.454093    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-gstzs_aad62943-cb5a-4e0d-af69-900892198939/main/0.log"
Jul 29 21:46:44.670953 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:44.670929    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftmq86_fc1371ff-4afa-469b-be32-f021cb7f8a70/main/0.log"
Jul 29 21:46:44.888680 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:44.888659    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-nqdwb_a5f09ebf-353b-41eb-b1e9-af28d67a43b1/main/0.log"
Jul 29 21:46:53.333889 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:53.333859    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6xrsn_4cb7aa5f-475b-4da5-b00a-1410e77e5f66/global-pull-secret-syncer/0.log"
Jul 29 21:46:53.519621 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:53.519589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cxrmk_16342c1c-a4d7-4067-8d0e-a53b55d46078/konnectivity-agent/0.log"
Jul 29 21:46:53.616895 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:53.616859    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-64.ec2.internal_e75f6cc257a11d76b0facbb4329f9f9e/haproxy/0.log"
Jul 29 21:46:59.187689 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:59.187661    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-7rqvx_048aaed8-8b03-40e2-ab2a-7fa6f2e62f99/kuadrant-console-plugin/0.log"
Jul 29 21:46:59.697283 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:46:59.697252    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-lkksg_4de8b85f-818f-4e3e-8bb1-fb19b637b0d6/manager/0.log"
Jul 29 21:47:01.728455 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:01.728419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-pj425_dd421a79-7fc7-46dc-9e78-3ac699f74657/cluster-monitoring-operator/0.log"
Jul 29 21:47:01.902370 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:01.902334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-tngbv_b9ce8dd9-b4c6-4bc0-a8b6-3ea22c749ee6/monitoring-plugin/0.log"
Jul 29 21:47:02.082816 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.082738    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sknft_474e50f6-bd18-434d-bc8f-256acc21866b/node-exporter/0.log"
Jul 29 21:47:02.113519 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.113500    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sknft_474e50f6-bd18-434d-bc8f-256acc21866b/kube-rbac-proxy/0.log"
Jul 29 21:47:02.148687 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.148670    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sknft_474e50f6-bd18-434d-bc8f-256acc21866b/init-textfile/0.log"
Jul 29 21:47:02.719896 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.719868    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-wkswh_764df4a6-3daf-4708-a92a-49c78f4191f3/prometheus-operator-admission-webhook/0.log"
Jul 29 21:47:02.844076 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.844053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69cbc65479-m6t74_5afca716-ca45-4f76-948b-9c13dcceedd1/thanos-query/0.log"
Jul 29 21:47:02.865526 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.865503    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69cbc65479-m6t74_5afca716-ca45-4f76-948b-9c13dcceedd1/kube-rbac-proxy-web/0.log"
Jul 29 21:47:02.891301 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.891278    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69cbc65479-m6t74_5afca716-ca45-4f76-948b-9c13dcceedd1/kube-rbac-proxy/0.log"
Jul 29 21:47:02.914044 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.914023    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69cbc65479-m6t74_5afca716-ca45-4f76-948b-9c13dcceedd1/prom-label-proxy/0.log"
Jul 29 21:47:02.936199 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.936183    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69cbc65479-m6t74_5afca716-ca45-4f76-948b-9c13dcceedd1/kube-rbac-proxy-rules/0.log"
Jul 29 21:47:02.978138 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:02.978080    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-69cbc65479-m6t74_5afca716-ca45-4f76-948b-9c13dcceedd1/kube-rbac-proxy-metrics/0.log"
Jul 29 21:47:03.562511 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.562474    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"]
Jul 29 21:47:03.563192 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.563168    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy"
Jul 29 21:47:03.563341 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.563194    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7882195f-a528-4807-a3d6-15c6e92f2578" containerName="istio-proxy"
Jul 29 21:47:03.563341 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.563209    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f123d55-3da5-48f2-a4ad-a76fd3286658" containerName="revoke-keys"
Jul 29 21:47:03.563341 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.563219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f123d55-3da5-48f2-a4ad-a76fd3286658" containerName="revoke-keys"
Jul 29 21:47:03.563513 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.563445    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f123d55-3da5-48f2-a4ad-a76fd3286658" containerName="revoke-keys"
Jul 29 21:47:03.567155 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.567136    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.574114 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.574094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wkbd4\"/\"default-dockercfg-m2tq2\""
Jul 29 21:47:03.574927 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.574906    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wkbd4\"/\"openshift-service-ca.crt\""
Jul 29 21:47:03.575048 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.574965    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wkbd4\"/\"kube-root-ca.crt\""
Jul 29 21:47:03.576938 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.576919    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"]
Jul 29 21:47:03.621012 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.620989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-lib-modules\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.621140 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.621019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-podres\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.621140 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.621058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-sys\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.621224 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.621138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-proc\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.621224 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.621169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2nv\" (UniqueName: \"kubernetes.io/projected/cb085489-14f8-4a1e-8e39-13c9245ac9d7-kube-api-access-6v2nv\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.721940 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.721913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-proc\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722056 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.721946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2nv\" (UniqueName: \"kubernetes.io/projected/cb085489-14f8-4a1e-8e39-13c9245ac9d7-kube-api-access-6v2nv\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722056 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.721973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-lib-modules\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722056 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.722003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-podres\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722056 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.722028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-proc\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722207 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.722061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-sys\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722207 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.722102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-sys\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722207 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.722128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-lib-modules\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.722207 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.722179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cb085489-14f8-4a1e-8e39-13c9245ac9d7-podres\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.731260 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.731244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2nv\" (UniqueName: \"kubernetes.io/projected/cb085489-14f8-4a1e-8e39-13c9245ac9d7-kube-api-access-6v2nv\") pod \"perf-node-gather-daemonset-n27ss\" (UID: \"cb085489-14f8-4a1e-8e39-13c9245ac9d7\") " pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:03.876839 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:03.876818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:04.011994 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:04.011967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"]
Jul 29 21:47:04.014206 ip-10-0-136-64 kubenswrapper[2579]: W0729 21:47:04.014181    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcb085489_14f8_4a1e_8e39_13c9245ac9d7.slice/crio-8815fafe88ebac5d5dd54c5a0cf750915c2fb94f28e3a7eab3e1070221b1b18c WatchSource:0}: Error finding container 8815fafe88ebac5d5dd54c5a0cf750915c2fb94f28e3a7eab3e1070221b1b18c: Status 404 returned error can't find the container with id 8815fafe88ebac5d5dd54c5a0cf750915c2fb94f28e3a7eab3e1070221b1b18c
Jul 29 21:47:04.251571 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:04.251504    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-l8cbz_c51b7261-61be-4338-a660-f6b2d192525d/networking-console-plugin/0.log"
Jul 29 21:47:04.465691 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:04.465660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss" event={"ID":"cb085489-14f8-4a1e-8e39-13c9245ac9d7","Type":"ContainerStarted","Data":"0735376594a2c29403415b929813374015624a554130344faf45eb4c37b900f8"}
Jul 29 21:47:04.465831 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:04.465705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss" event={"ID":"cb085489-14f8-4a1e-8e39-13c9245ac9d7","Type":"ContainerStarted","Data":"8815fafe88ebac5d5dd54c5a0cf750915c2fb94f28e3a7eab3e1070221b1b18c"}
Jul 29 21:47:04.465831 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:04.465726    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:04.484127 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:04.483757    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss" podStartSLOduration=1.483741436 podStartE2EDuration="1.483741436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:47:04.482644289 +0000 UTC m=+3573.799133203" watchObservedRunningTime="2026-07-29 21:47:04.483741436 +0000 UTC m=+3573.800230349"
Jul 29 21:47:05.389369 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:05.389339    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-65qpc_4be3e202-01fd-4674-b9cc-e8f095d16563/download-server/0.log"
Jul 29 21:47:07.142371 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:07.142345    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z9b6g_3b8528ab-0a79-4d17-807a-597976ead2f6/dns/0.log"
Jul 29 21:47:07.168569 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:07.168544    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z9b6g_3b8528ab-0a79-4d17-807a-597976ead2f6/kube-rbac-proxy/0.log"
Jul 29 21:47:07.221134 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:07.221117    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m4kxx_33fdf5c4-3b9d-4f59-92e4-b66c57f165ac/dns-node-resolver/0.log"
Jul 29 21:47:07.862030 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:07.861998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-s9brg_a54988dd-54c1-44d6-ac26-8e9964f9794a/node-ca/0.log"
Jul 29 21:47:09.064623 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:09.064593    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-wjm28_643b3181-bfa7-4674-9428-a23fba762430/kube-auth-proxy/0.log"
Jul 29 21:47:10.147123 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:10.147096    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xk52g_b691b5e0-c6cc-479d-a23c-e9e96fc40efb/serve-healthcheck-canary/0.log"
Jul 29 21:47:10.479641 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:10.479573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wkbd4/perf-node-gather-daemonset-n27ss"
Jul 29 21:47:10.646387 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:10.646358    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-m7wjk_4ffc6d67-25c5-4c0b-9d7e-693c2a790be1/insights-operator/0.log"
Jul 29 21:47:10.647846 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:10.647821    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-m7wjk_4ffc6d67-25c5-4c0b-9d7e-693c2a790be1/insights-operator/1.log"
Jul 29 21:47:10.669728 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:10.669705    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7pfpq_dd6703c2-da11-4da2-b67c-50513265ba18/kube-rbac-proxy/0.log"
Jul 29 21:47:10.691710 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:10.691693    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7pfpq_dd6703c2-da11-4da2-b67c-50513265ba18/exporter/0.log"
Jul 29 21:47:10.719166 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:10.719146    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7pfpq_dd6703c2-da11-4da2-b67c-50513265ba18/extractor/0.log"
Jul 29 21:47:14.047005 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:14.046973    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-n9dbs_3ca56f15-86bc-4c57-9025-b0dbfde96e35/manager/0.log"
Jul 29 21:47:14.089142 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:14.089121    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-57645d8769-n9dbs_3ca56f15-86bc-4c57-9025-b0dbfde96e35/copy-manifests/0.log"
Jul 29 21:47:14.646535 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:14.646507    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-wjm28_643b3181-bfa7-4674-9428-a23fba762430/kube-auth-proxy/0.log"
Jul 29 21:47:15.556866 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:15.556840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-64fb5f9c89-dg9k9_84445ff9-de69-45a0-9c38-31c539dfd9b1/manager/0.log"
Jul 29 21:47:16.494361 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:16.494334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68db9f4dd4-wjm28_643b3181-bfa7-4674-9428-a23fba762430/kube-auth-proxy/0.log"
Jul 29 21:47:20.684916 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:20.684838    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-knsqx_49d8915c-5299-497a-955c-a7b9bfcd1162/migrator/0.log"
Jul 29 21:47:20.711233 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:20.711209    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-knsqx_49d8915c-5299-497a-955c-a7b9bfcd1162/graceful-termination/0.log"
Jul 29 21:47:21.112454 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:21.112429    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2_c1034613-bed5-4ad8-908c-7aa2cdde9e1c/kube-storage-version-migrator-operator/1.log"
Jul 29 21:47:21.114104 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:21.114071    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-4xkw2_c1034613-bed5-4ad8-908c-7aa2cdde9e1c/kube-storage-version-migrator-operator/0.log"
Jul 29 21:47:22.219260 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.219221    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2mrqp_efdf74bf-a4a6-4b85-9aa6-39e1db215820/kube-multus/0.log"
Jul 29 21:47:22.248707 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.248677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-782xm_f2728239-662b-4632-acf7-8ed1b5287ca4/kube-multus-additional-cni-plugins/0.log"
Jul 29 21:47:22.272887 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.272867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-782xm_f2728239-662b-4632-acf7-8ed1b5287ca4/egress-router-binary-copy/0.log"
Jul 29 21:47:22.297225 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.297206    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-782xm_f2728239-662b-4632-acf7-8ed1b5287ca4/cni-plugins/0.log"
Jul 29 21:47:22.320421 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.320401    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-782xm_f2728239-662b-4632-acf7-8ed1b5287ca4/bond-cni-plugin/0.log"
Jul 29 21:47:22.343555 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.343536    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-782xm_f2728239-662b-4632-acf7-8ed1b5287ca4/routeoverride-cni/0.log"
Jul 29 21:47:22.368119 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.368100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-782xm_f2728239-662b-4632-acf7-8ed1b5287ca4/whereabouts-cni-bincopy/0.log"
Jul 29 21:47:22.394877 ip-10-0-136-64 kubenswrapper[2579]: I0729 21:47:22.394857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-782xm_f2728239-662b-4632-acf7-8ed1b5287ca4/whereabouts-cni/0.log"